Updates from CASU

CASU
Processing: Overview and
Updates for the VVV Survey
Nicholas Walton
Eduardo Gonalez-Solares, Simon
Hodgkin, Mike Irwin
(Institute of Astronomy)
Pipeline Processing Summary
Data organization (check images, headers, rename
files, ingest into raw data archive)
● Data processing (dark correction, linearity, flat field,
sky, destripe, stack jitters, cataloguing, astrometric and
photometric calibration)
● Produces one stacked pawprint image for each OB (+
associated confidence map and catalogue)
● Mosaic (tile) the stacked pawprint for each OB (+
associated confidence map and catalogue) [in
progress]
● Post processing checks
●
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
2
Processing Status: Mar 2012
●
●
Processing is proceeding nominally
P88: All data to 22 Feb 2012 reduced and will be released in
the next few days.
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
3
VISTA Survey Progress
http://casu.ast.cam.ac.uk/vistasp/overview
as of 14 Dec 2011
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
4
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
5
VVV data volumes are significant
VVV
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
6
Turning
VISTA focalthis
plane…
the VISTA Focal Plane
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
7
… to this … e.g Orion
M78 as seeing by VIRCAM in JHK
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
8
Pipeline Overview
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
9
… generating Science Data Products
●
products consist of:
—
—
—
—
—
—
●
●
calibrated single exposure images
shifted ”average” stack frames (pawprints) + conf maps
calibrated stacked pawprint catalogues
filled area tile images + confidence maps
calibrated tile object catalogues
sky background images, flats, darks, bad pixel mask
science products are MEF (multi-extension FITS) files
(images Rice-compressed)
all QC parameters are stored in MEF headers
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
10
Image processing steps:
removing the instrumental signature
•
•
•
•
•
•
•
Reset correction (debias - inline)
Dark correction
Linearity correction
Flat field correction
Sky background correction
Destripe - controller level pickup
Crosstalk, persistence and fringing corrections
are not necessary
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
11
VISTA data flow - I
●
●
●
●
●
●
●
●
raw data transfers on USB disk (Rice-compressed MEFs)
ingest & verification → raw data archive
create off-line tape backups
update calibration files monthly (flats, linearity, masks)
parallel nightly processing at OB-level (darks updated)
—
stacked pawprint images instrumental signature removed
—
catalogue generation from pawprint images & conf maps
—
astrometric & photometric calibration
header updates → pawprint OB-level science products
check derived QC info & sample of images
processing web page updates and progress tracking:
http://casu.ast.cam.ac.uk/surveys-projects/vista
12
Nightly Quality Control
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
13
VISTA data flow - II
●
●
●
●
●
detector level monthly photometric zpt updates
— illumination correction tables
mosaic OB-level tile image and create confidence map
tile image cataloguing (Tangent Plane projection)
— nebulosity filter 6 component pawprints
— mosaic and correct for sky levels and distortion
— generate tile catalogue
— grout tiles to fix PSF and detector zpt variations
check derived QC info & sample of images (cf. OB grade)
ingest to post-processing database enables checks:
— FITS header contents, file size, provenance and calibration files,
exploration of long-term trends, survey progress, data access
http://casu.ast.cam.ac.uk/vistasp/imgquery/search
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
14
Survey progress
overview
QC plots
summarise:
astrometry; seeing;
stellar ellipticity;
sky brightness;
magnitude zeropoint trends
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
15
Mag ZP Monitoring
Silvered mirror
Aluminised mirror
Photometric ZP variation
Cryostat window
cleaned
P85
22 Mar 2012
P86
P87
Nic Walton - CASU Updates - VVV @ Vina del Mar
P88
16
Tile Creation
1 tile = 6 stacked paws each with 16 detector level images
Tiles and confdence maps
@ 1 GB ~ 100GB
(25GB compressed)
Tile
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
Confidence Map
17
Tile Generation & Corrections
http://casu.ast.cam.ac.uk/surveys-projects/vista/technical/tiles
A number of subtleties in extracting the catalogues from the files
● PSF variations across the 96 detectors in each time
● PSF variations increased during poorer seeing periods
● GROUTING removes a number of these issues
● For any object computes a photometric correction by reference to the
aperture correction at that object weighted by the confidence map
●
Example magnitude and colour-colour diagrams for
tiles before (left) and after (right) grouting
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
18
Astrometric Calibration
2MASS - VISTA
WCS - ZPN projection
NB. tiles are TAN projection
Linear solution
per detector
rms < 100 mas
Tabulated systematics
from stacked residuals
sys < 25 mas
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
19
Photometric Calibration
●
●
colour equations
— convert 2MASS to instrumental system
2MASS s:n > 10 in J,H,Ks and
—
—
—
●
NIR ~100-1000 “standards” per pointing
—
●
●
0 < (J-Ks) < 2 & 0 < (J-Ks)_o < 1 (extinction corr)
0 < (J-Ks) & (J-Ks)_o < 1 & (J-Ks) < 0 (update extcorr)
no restriction
required to be stellar and unsaturated
Zpt + error per pointing; can compare with FS fields
monitor long term Zpt behaviour
—
—
average monthly detector Zpt offsets
illumination corrections
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
20
VIRCAM Stripes
cleaning and correcting the data
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
21
Removing the (time varying)
background sky …
●
●
●
●
●
Tilesky – double pass combination of all observations in
tile(s)
Pawsky – single pass combination of all observations in a
pawprint with object masking iterated ‘dynamically’
Pawsky with object mask – as above, but the mask is defined
beforehand using e.g. deep stacked tiles
Offset sky – use a sky taken nearby (spatially & temporally)
Pawsky and “half” tilesky - minus ****
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
22
Before And After Background
Correction
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
23
Flat Field Holes
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
24
Sky Frame Blobs (not stars)
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
25
Software modules
●
nebuliser
—
—
●
mosaicer
—
●
CASU tiling software developed for VISTA
grouter
—
●
removes complex background variations
enhanced object detection & parameterisation
applied to tile catalogues to remove the effect of PSF
variations and photometric throughput (+ MJD column)
psf’ers – under development …
—
—
automatically generates detector-level PSFs
and performs PSF photometry
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
26
Background Subtraction
Nebulosity filter applied to stacks before mosaicking
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
27
Background Subtraction
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
28
Background Subtraction
example: use of nebuliser – M17/ K-band
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
29
Improved object detection
after removing background nebulosity
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
30
Issues with tiles
●
●
●
●
●
●
Imperfect sky subtraction pawprint matching : low
level discontinuous artifacts
Variable PSF across single pawprints : each detector
has different PSFs
Variable seeing conditions : each pawprint has
different PSFs
Variable extinction during the observations :
variation of ZP over tile
Astrometric distortion : photometric distortion
Interpolation options : varying correlated noise
patterns
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
31
PSF in each detector
... 96 different PSFs go into a tile ...
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
32
PSF fitting
Moffat fit is a good representation
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
33
Pixel Scale Variation
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
34
Photometric Distortion over a Tile
Inherent illumination correction
22 Mar 2012
Tile exposure map
Nic Walton - CASU Updates - VVV @ Vina del Mar
35
CASU Software Release
http://casu.ast.cam.ac.uk/surveys-projects/software-release
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
36
CASU Software Release: imcore
http://casu.ast.cam.ac.uk/surveys-projects/software-release
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
37
Known Issues
http://casu.ast.cam.ac.uk/surveys-projects/vista/technical/known-issues
●
●
●
●
●
●
●
●
●
●
●
Bad pixels in detector number 1
Point-like objects in flat and sky frames
Detector 16 variable QE
Bad channel in detector 6
Filter wheel out of position
Detector #4
Spurious detections around bright stars
Images with large ellipticities
Persistance
Tile images variable depth
Aperture2 fluxes – now fixed (Aper3 is default)
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
38
CASU – new features under
development
●
Improved PSF Fitting
—
●
●
●
Tailored for high density fields, thus VVV
Improved handling of multi-epoch data
Improved data transfers over the EVALSO link
Implementation of CASU pipeline to run in 'quick
look' mode at Paranal
—
Generation of Reflex workflows for processing VIRCAM
data
Currently using Version 1.2 of the processing software (since 1 May 2011)
Always check the change log at
http://casu.ast.cam.ac.uk/surveys-projects/vista/data-processing/version-log
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
39
http://casu.ast.cam.ac.uk
22 Mar 2012
Nic Walton - CASU Updates - VVV @ Vina del Mar
40