Our goal is to extract the lightcurve for any star within the PBT field of view, and to explore it for evidence of periodic or other variability. The principle steps in this procedure are as follows.
The first database required will allow us to associate observation times and camera number with each unique PBT image. Thus there will be one record per image containing:
\"PbtImage\" table:
{{{
- image_id (unique key!) (INT) # see below
- camera_id (smallINT) # 1,0,-1 corresponding to 126, 128 and 129 respectively
- spocdate (date) #date in image header
- spoctime (time) # time in image header
- raid_time (time) # time decoded from file creation time
- obs_jd (INT) # julian date (integer part)
- obs_fjd (INT) # fractional julian date * 10^6
- exp_time (INT) # 30
- data_disk (STRING) # raid5,raid15
- data_folder (STRING) # starlight/<yyyymo>/<yyyymodd_cam>
- data_file (STRING) # <cam_yyyymodd_hhmiss>.fit
}}}Where the imageid is created from the camera number, the integer part of the Julian date and the first -five- digits of the fractional part of the julian date, -i.e.- imageid=cjjjjfffff. Note the date and time used in the imageid are the 'obs' times, i.e. the raidtime - 37secs.
-Example:- The image 12920090109182930.fit has an image_id -1484177006
PostgreSQL is used to create the tables into which the files shall be read.
{{{
#create image table
psql PBT -U rkj -c \" \\
CREATE TABLE PBTimage ( \\
image_id int, \\
camera smallint, \\
spocdate date, \\
spoctime time, \\
raid_time time, \\
julian_date smallint, \\
fractional_time int, \\
exposure_time smallint, \\
data_disk text, \\
data_folder text, \\
data_file text \\
);\"
##read data into image table
cat image.csv | psql PBT -U rkj -c \"COPY PBTimage FROM STDIN WITH CSV\"
}}}The second database will contain all the photometry associated with each image, but the key identifier will be the object itself, as identified from the USNO catalogue using stilts. Thus there will be one record per object per PBT image on which it was identified, and it will contain:
"PbtPhotoObj" table:
{{{
- catalog_id (INT)
- image_id (referring to PbtImage) (INT)
- RA (smallint) ##have used mod(360)
- DEC (samllint) ##before converting to integers
- image_x (real)
- image_y (real)
- mag (real) ##added 21, then multiplied by 1000
- mag error (real) ##multiplied by 1000
- flag (smallINT)
- fwhm (smallint) ##multiplied by 10
- separation (smallINT)
}}}The data required to populate the second database will be achieved using stilts,
csj has drafted a script pbt_stilts to achieve the first stage. However, we have not yet established how we are to insert each nights data into the database, nor have we established the format for the database, although we are likely to require that it to be SQL compatible. -
{{{
#create object table
psql PBT -U rkj -c \" \\
CREATE TABLE PBTphot ( \\
number int, \\
image_id int, \\
ra smallint, \\
dec smallint, \\
x_position smallint, \\
y_position smallint, \\
mag smallint, \\
mag_error smallint, \\
flags smallint, \\
fwhm smallint, \\
separation smallint \\
);\"
##read data into object table
cat $1_usno.csv | psql PBT -U rkj -c \"COPY PBTphot FROM STDIN WITH CSV HEADER\"
}}}For a given catalogue star.
For each camera
Open data file and write header: -(see below regarding format for this data)-
{{{ Catalogue identifier Camera id Number of comparisons Comparison Identifiers Mean catalogue DM }}}
For each frame
compute mean DM between target and comparison stars
compute mean DM error
save as
{{{ Obs time (JD) : Mean DM : DM Error (DME) }}}
*** Close data file
The previous steps should, if applied to the entire database, yield a number of files (from several hundred to ten thousand) containing time-series differentical photometry for an individual star in the PBT field. There should be three files for each star, one for each camera. Once we understand the systematics better, it may be possible to combine data from different cameras.
The next step is to examine the data
talk to gavin about tools for doing frequency analysis. Ask about format for input files.USNO-B is an all-sky catalogue that presents positions, proper motions, magnitudes in various optical passbands, and star/galaxy estimators for 1,042,618,261 objects. It is believed to provide all-sky coverage, completeness down to V = 21, 0.2 arcsecond astrometric accuracy at J2000, 0.3 magnitude photometric accuracy in up to five colors, and 85% accuracy for distinguishing stars from non-stellar objects.
It has been identified as the primary source for identifying objects in the PBT survey.
The local version of the USNO catalogue on arpc55 is located at
{{{ /data/usno_catalogue/usno_RA_DEClabelled }}}topcat is an interactive graphical viewer and editor for tabular data. Its aim is to provide most of the facilities that astronomers need for analysis and manipulation of source catalogues and other tables, though it can be used for non-astronomical data as well. It understands a number of different astronomically important formats (including FITS and VOTable) and more formats can be added.-
This tool is useful for examining the positions of objects in -.phot- files and comapring them with standard reference catalogues.
stilts is a set of command-line tools for processing tabular data. It has been designed for, but is not restricted to, use on astronomical data such as source catalogues. It contains both generic (format-independent) table processing tools and tools for processing VOTable documents. Facilities offered include crossmatching, format conversion, format validation, column calculation and rearrangement, row selection, sorting, plotting, statistical calculations and metadata display. Calculations on cell data can be performed using a powerful and extensible expression language.
This tool has been identified as the optimal means of cross-correlating PBT world coordinates with the catalogue, and for extracting and storing date from -.phot- files into either VOTable format or FITS binary table format.
For correlating PBT data with the USNO-B catalogue, a number of search criteria are required. These have to be
sufficiently to allow stilts to find a good match. Recall that the a) the PBT image scale is extremely large,
(?? arc seconds per pixel) b) image smearing increases towards lower declination and c) errors in the astrometric solution will be compounded towards image corners. The code "params=300" means that a tolerance up to 300 arc seconds is necessary, but the user should be cautious about false matches.
The use of stilts to write a VOTable output has not been completely tested. We still have questions regarding timing, and the required format of the output database.
{{{
stilts tpipe \\
in=$1_usno.tx1 ifmt=ascii \\
cmd='colmeta -name USNO_ID 1' \\
cmd='colmeta -name B_USNO 4' \\
cmd='colmeta -name RA_pbt -units \"pixels\" 5' \\
cmd='colmeta -name DEC_pbt -units \"pixels\" 6' \\
cmd='colmeta -name X -units \"pixels\" 7' \\
cmd='colmeta -name Y -units \"pixels\" 8' \\
cmd='colmeta -name MAG_pbt -units \"mag\" 9' \\
cmd='colmeta -name MAGERR_pbt -units \"mag\" 10' \\
cmd='colmeta -name FLAGS 11' \\
cmd='colmeta -name FWHM -units \"pixels\" -desc \"full width half maximum\" 12' \\
cmd='colmeta -name E -desc \"ellipticity\" 13' \\
cmd='colmeta -name SEP -units \"arcsec\" -desc \"great circle separation between USNO and PBT positions\" 14' \\
cmd=\"setparam -desc \\\"pbt camera\\\" CAM ${cam}\" \\
cmd=\"setparam -desc \\\"obs year\\\" YEAR ${yy}\" \\
cmd=\"setparam -desc \\\"obs day\\\" DAY ${od}\" \\
cmd=\"setparam -desc \\\"obs time (UT)\\\" TIME ${th}:${tm}:${ts}\" \\
cmd=\"setparam -desc \\\"Julian Date of Obs (int)\\\" JD1 ${jd}\" \\
cmd=\"setparam -desc \\\"Julian Date of Obs (frac)\\\" JD2 ${fd}\" \\
omode=out out=$1_usno.vo ofmt=votable-binary-inline
}}}The code below combines the stilts code to match the images and the catalogue with code to calculate the julian date, fractional time, data folder, image id, etc.
{{{
#! /bin/bash
# Julian Day Number from calendar date
date2julian() # year month day
{
d2j_year=$1
d2j_month=$2
d2j_day=$3
## Calculate number of months from March
d2j_tmpmonth=$((12 * 10#$d2j_year + 10#$d2j_month - 3))
#echo $d2j_tmpmonth
d2j_tmpyear=$((10#$d2j_tmpmonth / 12))
#echo $d2j_tmpyear
date2julian=$(( (734 * 10#$d2j_tmpmonth + 15) / 24 - 2 * 10#$d2j_tmpyear + \\
10#$d2j_tmpyear/4 - 10#$d2j_tmpyear/100 + 10#$d2j_tmpyear/400 + 10#$d2j_day + 1721119 ))
echo $date2julian
return 0
}
#ordinal date
odate()
{
declare -i year month day odate
declare -a days
local year=$1 month=${2/0/} day=${3/0/} odate
days=( 0 31 59 90 120 151 181 212 243 273 304 334 )
let odate=days[month-1]+day
if (( (year%4 == 0) && (month > 2) ))
then
let odate+=1
fi
echo $odate
return 0
}
## convert time to fractional day
fracday()
{
#declare -i year month day odate
declare -a th tm ts fd
local th=$1 tm=${2} ts=${3} fd
fd=$(bc << EOF
scale=5
ms = $ts/60.
ms = ms + $tm
hms = ms/60.
hms = hms + $th
hms/24.
EOF
)
echo $fd
return 0
}
# convert camera numbers to single digit
camera()
{
case $cam in
126 ) echo \"1\";;
128 ) echo \"2\";;
129 ) echo \"3\";;
* ) echo \"error\";;
esac
}
cam=${1:0:3}
yy=${1:4:4}
mm=${1:8:2}
dd=${1:10:2}
th=${1:13:2}
tm=${1:15:2}
ts=${1:17:2}
od=`odate ${yy} ${mm} ${dd}`
jd=`date2julian ${yy} ${mm} ${dd}`
fd=`fracday ${th} ${tm} ${ts}`
n=`camera ${cam}`
}}}In order to have an 'obs' time for each image we need to adjust the raid time by subtracting 37 seconds (the time taken to create and read out the file). We also want to edit the time and date so that they are both integers and as small as possible in order to reduce the size of the database.
{{{
fd2=${fd:1:7} ###remove decimal point
fd3=`echo \"($fd2 -42 )\" | bc` ###subtract 37secs to give 'obs' time
fd4=`printf %0.f $fd3` ###round to nearest second
if (( $fd4 < 0 )) ###if less than 37secs after midnight
then
fd2=$(( 100000 - ( 42 - 10#$fd2 ) )) ###change time
jd=$(( $jd - 1 )) ###change day
else fd2=$fd4 ###otherwise obs time equals rounded time from above
fi
md=$(( $jd-2450000 )) #remove unnecessary digits from julian date
image_id=$n$md$fd2 #create image id
}}}Stilts script to match -.phot- file with the USNO catalogue and then edit the output file.
{{{
stilts tmatch2 \\
ifmt1=ascii \\
in1=/data/usno_catalogue/usno_ncp_index \\
icmd1='delcols \"2 6 7 8\"' \\
ifmt2=ascii \\
in2=$1.phot \\
icmd2='delcols \"1 6 7 10 11 14 15\"' \\
values1=\"\\$2 \\$3\" suffix1=_usno \\
values2=\"\\$1 \\$2\" suffix2=_pbt \\
matcher=sky params=300 join=1and2 \\
omode=out out=$1_usno.tx1 ofmt=ascii
stilts tpipe \\
ifmt=ascii \\
in=$1_usno.tx1 \\
cmd='delcols \"2 3 4\"' \\
cmd='addcol -after col1 image_id \\\"'$image_id'\\\"' \\
cmd='replacecol Separation toInteger(Separation)' \\
cmd='replacecol col4_pbt toInteger(col4_pbt)' \\
cmd='replacecol col5_pbt toInteger(col5_pbt)' \\
cmd='replacecol col2 (((col2+180)%360)-180)*100' \\
cmd='replacecol col3_pbt (((col3_pbt+180)%360)-180)*100' \\
cmd='replacecol col2 toInteger(col2)' \\
cmd='replacecol col3_pbt toInteger(col3_pbt)' \\
cmd='replacecol col8 (col8+21)*1000' \\
cmd='replacecol col8 toInteger(col8)' \\
cmd='replacecol col9 (col9)*1000' \\
cmd='replacecol col9 toInteger(col9)' \\
cmd='replacecol col13 (col13)*10' \\
cmd='replacecol col13 toInteger(col13)' \\
omode=out out=$1_usno.csv ofmt=csv
rm $1_usno.tx1
}}}We now need to create the data for the image database, using the .FITS files.
{{{
HEADAS='/home/csj/Applications/heasoft-6.9/i686-pc-linux-gnu-libc2.5/'
export HEADAS
source /home/csj/Applications/heasoft-6.9/i686-pc-linux-gnu-libc2.5//headas-init.sh
#extract time and date from image header
frame=$data_folder/$data_file
spocdate=`fkeyprint ${frame} date-obs | grep \"DATE-OBS=\" | sed -e 's/DATE-OBS= // ' | sed -e s/\\'//g `
spoctime=`fkeyprint ${frame} time-obs | grep \"TIME-OBS=\" | sed -e 's/TIME-OBS= // ' | sed -e s/\\'//g `
echo 'spocdate' $spocdate
echo 'spoctime' $spoctime
raid_time=$th':'$tm':'$ts
echo 'raid time' $raid_time
#create csv
echo $image_id,$n,$spocdate,$spoctime,$raid_time,$md,$fd2,$exptime,$data_disk,$data_folder,$data_file >image.csv
}}}