PBT_data_reduction

'''The following scripts run on each control machine:''' '''runstarlight.py:''' Is run on waking from hibernation by WakeupOnStandBy software. Starts the starlight software. Takes exposures until a specified time. Closes down starlight software. Sets WakeupOnStandBy to wake from hibernation at a specified time. Sets the Computer to Hibernation.

'''wolspoc1.py, wlspoc2.py, wolspoc3.py:''' Sends a magic packet to a specific control computer, waking it from hibernation so it can be controlled via e.g. VNC.

'''pbtsumimg''' sums a series of frames to a master using fartih and renames the summed image file as sum.fit.

'''pbtfits2fitsimg Converts unreadable files to ones that can be viewed using Gaia. Name IMG*.FIT changed to *.fit

'''The following command scripts may be found in /home/csj/Projects/polar_bear/'''

'''pbt_observe:''' sets up 3 vncviewer windows, one for each control computer, in order to control the cameras directly.

'''pbtimgname:''' renames all IMGnnnnn.FIT files to cccyyyymmddhhmmss.fit pbtimgname could be made redundant by automating the filenaming on the control machines. '''pbtclean''': moves raw cccyyyymmddhhmmss.fit files containing stars to /starlight/yyyymmddccc/,

'''pbt_phot:''' complete astrometry and photometry reduction for all images for all cameras for one night

'''pbt_extract:''' extracts light curves for all catalogued stars (one curve per star) from all frames for one night and one camera

'''phot_merge:''' (c++) reads the photometry output, compares positions with a catalogue, identifies 100 brightest stars, computes and applies a zero-point correction and ouputs the resulting data (x, y, ra, dec, mag, magerr, diam) as well as the catalogue information for identified stars.

pbtsphtrig.ots: a spread sheet used to extract various quantities for an image from its astrometry

photmerge.c++ : c++ source code for photmerge","",0,"","",0,"utf-8","" 2830,0,"","== Data Preprocessing ==

This job reads the file creation time for each image and creates a new filename in the form:

where ccc = camera number, yyyy = year, mm = month, dd = day, hh = hour, mm = minutes, ss = seconds.

pbt_imgname could be made redundant by modifying runstarlight.py to name images as they are recorded

This job reads through every file and identifies the number of stars visible on each image. The following files are deleted. All files taken during "daylight" hours (the script currently has to be told when these change). All files with fewer than c. 100 stars are deleted. All files with more than c. 15000 stars are deleted -- these are files with a fixed pattern noise obtained either when the cameras were switched off or when they were exposed in daylight and hence completely saturated.

deleted files are currently not deleted, but moved to a folder /starlight/spoc''n''/bucket/ until pbt_clean is proven to be working optimally.

** BUG ** pbt_clean currently puts all frames from the same calendar date into the same folder. This applies to dates since sometime in May. The original design was to put all frames from the same night into the same folder, with the calendar date corresponding to the date on which the night started ** Needs fixing **

Data may be easily inspected using any image viewing programme which recognises FITS file formats. The most useful are those designed for working with astronomical images including

It is worth checking random images through the night to check that the pre-processing was successful. It is also worth updating the observing log ''regularly'', including any helpful comments about observing conditions and so on. This provides an extremely valuable record when it comes to process the data and assess photometry extracted many months subsequently.

This can be saved as xhtml and viewed online -- e.g. [[http://www.arm.ac.uk/~csj/polarbear/pbtdata_checklist.xhtml |PBT Observing Log]]

To Be Done The observing log should be updated automatically by the various tasks it reports The xhtml version needs to be automatically updated from the base version

Calibration Frames

Bias Frames

Bias frames allow the "zero-point" of each CCD pixel to be established and subtracted from each science frame. They are normally frames of zero exposure time obtained in completely dark conditions. For PBT there are two complications. 1) The current software only allows a minimum exposure time of 1/1000 s. 2) The cameras are always open to the sky. To obtain bias frames with PBT:

i) Cover the telescope enclosures with a secure opaque material. Avoid direct sunlight as the cameras are sensitive in the infrared. ii) Obtain several (>>10) exposures in each camera with an exposure time of 1/1000s iii) Move the images for camera ccc to a (new) folder /starlight/biasyyyymmddccc/

Flat Fields

Flat fields are images of a uniformly illuminated flat object which enable variations in camera sensitivity to be removed. These variations include pixel-to-pixel variations in CCD sensitivity due to imperfection inthe CCD manufacture ''and'' large-scale variations due to the camera optics. In the case of PBT, [[http://en.wikipedia.org/wiki/Vignetting |vignetting]] causes severe loss of sensitivity at the periphery of the image.

The normal way to obtain a flat-field is to take an image of the twilight sky or a white screen inside the dome. PBT only has the twilight sky option. Since the field of view of each camera is large (10 degrees), even the twilight sky is not perfect (there is a large brightness gradient from one side of the sky to the other after the Sun has gone down). The presence of clouds and/or stars renders a flat-field image useless. So does the presence of saturated pixels. So it is necessary to obtain the flat fields while the sky is still bright enough that no stars are visible, but not so bright that the images are saturated.

Warning: On 2009/02/10 we experienced problems with flat-field images for camera 126 (spoc1). The images remain saturated long
after the other cameras have reached astronomically low light levels, and there is severe vertical banding on the images. With reducing light levels, the saturation cleared from the center of the image, but persisted at the edges. With very low light levels, the camera returned to normal. It is not clear whether this is a persistent or intermittent problem. On 2009/02/13 -- problem persists, though flat taken on uniform cloud at low light levels might be usable ''in extremis''.

Calibration Frame Processing

Bias frames were taken in early February 2009. To create master bias frames, ftools was used to sum images and then divide the summed image by a constant, N. Within the set of bias frames taken on 2009-02-09, some had exposure time 0.001, others 0.002 and others 0.5. In the set taken 2010-02-13, all had exposure time 0.1. For 1261, 128 and 129, the higher exposure times were ignored. These master biases include only exposure times of 0.001. For 1262, only exposure times of 0.1 were included.

{| border=\"1\"
|+ 
'''Data on Master Bias Frames'''

!Camera Number for Which Bias Was Made
!Name of Master Bias
!Date and Time Frames Taken
!Exposure Times of Frames Used 
!Number of Frames Used (N)
!Mean Pixel Value 
!Standard Deviation of Pixel Value
|-
| align=\"center\" | 126
| align=\"center\" | bias126_20090206_mean.fit
| align=\"center\" | 2009-02-06 @ 14:20-14:52
| align=\"center\" | 0.001
| align=\"center\" | 162
| align=\"center\" | 652.327
| align=\"center\" | 9.828
|-
| align=\"center\" | 126
| align=\"center\" | bias126_20090213_mean.fit
| align=\"center\" | 2009-02-13 @ 18:01-18:09
| align=\"center\" | 0.1
| align=\"center\" | 204
| align=\"center\" | 6505.06
| align=\"center\" | 593.5
|-
| align=\"center\" | 128
| align=\"center\" | bias128_20090206_mean.fit
| align=\"center\" | 2009-02-06 @ 14:20-14:52
| align=\"center\" | 0.001
| align=\"center\" | 129
| align=\"center\" | 2285.23
| align=\"center\" | 10.12
|-
| align=\"center\" | 129
| align=\"center\" | bias129_20090206_mean.fit
| align=\"center\" | 2009-02-06 @ 14:20-14:52
| align=\"center\" | 0.001
| align=\"center\" | 115
| align=\"center\" | 2168.47
| align=\"center\" | 22.19
|}

Flat fields were taken in early February 2009 and 2010. To choose suitable flat frames to make the master, every fifth frame in the set of flats was examined. Data was tabulated in a table with headers "Image Number", "Time", "Exposure Time", "", "Polaris max", and "Comments". Frames with between 11000 and 59900 were used (depending largely on the range for each set of data).

{| border=\"1\"
|+ 
'''Data on Flat Fields'''
!Camera Number for Which Flat Was Made
!Name of Mean Flat
!Name of Final Flat
!Date and Time Frames Taken
!Exposure Times of Frames Used 
!Number of Frames Used (N)
!Mean Pixel Value 
!Standard Deviation of Pixel Value
!Max Pixel Value
|-
| align=\"center\" | 126
| align=\"center\" | meanflat126_20100211.fit
| align=\"center\" | smoothnorm126.fit
| align=\"center\" | 2010-02-11 @ 18:00-18:03
| align=\"center\" | 0.01
| align=\"center\" | 41
| align=\"center\" | 12513.74
| align=\"center\" | 1294.283
| align=\"center\" | 28955.299
|-
| align=\"center\" | 128
| align=\"center\" | meanflat128_20090210.fit
| align=\"center\" | smoothnorm128.fit
| align=\"center\" | 2009-02-10 @ 17:35-17:47
| align=\"center\" | 0.01
| align=\"center\" | 136
| align=\"center\" | 24893.67
| align=\"center\" | 1985.530
| align=\"center\" | 29262.236  
|-  
| align=\"center\" | 128
| align=\"center\" | meanflat128_20100211.fit
| align=\"center\" | N/A
| align=\"center\" | 2010-02-11 @ 17:39-17:49
| align=\"center\" | 0.01
| align=\"center\" | 86
| align=\"center\" | 26744.99
| align=\"center\" | 3441.227
| align=\"center\" | 32718.691
|-
| align=\"center\" | 129
| align=\"center\" | meanflat129_20090210.fit
| align=\"center\" | smoothnorm129.fit
| align=\"center\" | 2009-02-10 @ 17:32-17:44
| align=\"center\" | 0.01
| align=\"center\" | 141
| align=\"center\" | 26479.12
| align=\"center\" | 3386.300
| align=\"center\" | 33086.387
|-
| align=\"center\" | 129
| align=\"center\" | meanflat129_20100211.fit
| align=\"center\" | N/A
| align=\"center\" | 2010-02-11 @ 17:46-17:51
| align=\"center\" | 0.01
| align=\"center\" | 45
| align=\"center\" | 19018.61
| align=\"center\" | 3132.683
| align=\"center\" | 25792.689
|}

The bias frame created for camera 129 appeared as a dark frame with randomly scattered bright pixels at an average of about 30 counts above the background. On further examination, it became evident that these bright pixels appeared at the same x and y coordinates for camera 129 on the individual bias frames and on the individual flat field frames. It was decided that the bias should continue to contain these 'bad' pixels. Ideally, bias frames should be taken much more frequently, allowing a clearer view of whether problems like these are transient. [[File:/home/cmf/Documents/snapshot9.jpg]]

fimgstat was used to find the mean and maximum pixel value and the standard deviation of pixel values in the frame. Mean flats chosen to create final flats were: meanflat12620100211.fit, meanflat12820090210.fit, and meanflat129_20090210.fit. To create normalised flat fields, the mean flats were divided by their maximum pixel values. Smoothed flat fields were created using the normalised flat fields and a Gaussian function with a full width at half maximum of 10.

bias12620090206mean.fit smoothfinalflat126.fit meanflat12620100211.fit normflat12620100211.fit

bias12820090206mean.fit smoothfinalflat128.fit meanflat12820090210.fit
meanflat12820100211.fit normflat12820090210.fit normflat12820100211.fit

bias12920090206mean.fit smoothfinalflat129.fit meanflat12920090210.fit meanflat12920100211.fit normflat12920090210.fit normflat12920100211.fit

PBT Data Processing

Data Acquisition and Processing Scripts

Calibration Frames

Bias Frames

Flat Fields

Calibration Frame Processing