pro niscal, SOURCEDIR=sourcedir, CFILE=cfile, HELP=help, clean=clean ;============================================================================= ; ABSTRACT ; Top level procedure for NIS calibration. ; Calibrate nis spectral data from FITS format files ; This procedure finds and parses a configuration file (*.conf usually) that ; contains the various parameters to be passed to main calibration procedure ; called nis_proc. ; ; INPUTS ; ; SOURCEDIR - directory where NISCAL files are located. Defaults to ; '/project/near/ST/NIS/zen/nis_cal/' or whatever is ; hardcoded in the program body. ; cfile - name of configuration file to be read. This file contains info ; for passing to nis_proc ; clean - toggle is this a recalibration that requires cleanout of old data. ; HELP - entering /help will give information about using nis_cal ; ;The configuration file loooks like this (the actual file is aligned left ; without the semicolon or space): ; ;----------.conf file begins-------- ; ## This is a NISCAL configuration file: 7 Oct. 1998 version ; ## Please modify the parameters in the first column only, even ; ## if the entry should be blank ; /ext2/NIS/FLIGHT_DATA/1998/ ###DIRECTORY_OF_RAW_NIS_INPUT_DATA ; /ext2/NIS/FLIGHT_DATA/1998/ ###DIRECTORY_FOR_CALIBRATED_NIS_OUTPUT_DATA ; /dir/dir/list.data.test ###LIST_OF_RAW_NIS_FILENAMES (w. directory) ; /dir/dir/list.dark.test ###LIST_OF_RAW_NIS_DARKS (w. directory) ; 4 ###NISCAL1=CalibrationMode(0-5) ; 1 ###NISCAL2=DarkMode(0-8) ; 0 ###NISCAL3=NoiseMode(0-1) ; 0 ###NISCAL4=RadMode(0-1) ; 1 ###NISCAL5=VERBOSE(0-1) ;----------.conf file ends---------- ; ; Note that the file lists can be in another directory than the input data, ; and that their directory structure needs to be included in their ; parameters. ; ; CONFIGURATION FILE INPUTS ; !!These inputs will be called by the program and sent to nis_proc ; DIRECTORY_OF_RAW_NIS_INPUT_DATA - directory where input files are ; located ; DIRECTORY_FOR_CALIBRATED_NIS_OUTPUT_DATA - output directory ; LIST_OF_RAW_NIS_FILENAMES - File list, single column, of ; Uncalibrated NIS FITS ; files ; LIST_OF_RAW_NIS_DARKS - File list, single column, of Uncalibrated NIS FITS ; files for sequences of dark-only observations, ; for DarkMode 1 (bracketing darks) ; CalibrationMode - calibration mode from niscal ; 0 = Averaged raw spectra in DN, coverted to Floating point ; 1 = (spectra - dark) in DN ; 2 = fully corrected DN ; 3 = radiance ; 4 = I over F ; 5 = "The works" loops through all cal modes ; DarkMode - Source of Dark for calibration ; 0 = (default) linear interpolation from interleaved ; darks surrounding data spectra (with a sum or ; average dark for the spectra at each end. ; 1 = Spectra are non-interleaved. Darks are derived ; from 2 other dark-only sequences (usually ; preceeding and following spectra sequence) ; DARKS ARE SHUTTER-CLOSED ; 2 = Spectra are non-interleaved. Darks are derived ; from dark-only sequences LOOKING INTO ; SPACE. ; 3 = Dark subtraction SKIPPED (usually because it was ; done externally). This option will almost ; certainly crash on an interleaved dark sequence. ; 4 = Dark from a single reference dark file: ; Cc_300dark_mean.dat ; 5 = Darks may or may not be interleaved. Abundant space ; darks are found looking into space on either ; side of the asteroid (as in mirror scans across ; the asteroid. Dark subtraction is perfomed by ; fitting space background across the sequence and ; subtracting. ; 6 = Same s 5 but the fightis "tighter" to the dark ; window. No data is dropped to shrink the dark ; window. Better for sequences that have only one ; or two frames looking into space ata time. ; 7 = Same as mode 5, but including corrected shutter-closed darks ; 8 = same as mode 6, but including corrected shutter-closed darks ; NoiseMode - how spectral noise is determined "err" is used instead of ; "noise" throughout these procedures for programming ; simplicity ; 0 = noise for heach nis channel is determined from ; the standard deviation of all dark ; spectra in sequence (Default) ; 1 = initial noise predetermined from st. devs. from ; a file of 300 dark spectra taken on an early ; in-space NIS calibration run ; RadMode - select radiometric correction table to be used ; 0 = Radiative Transfer Model applied to gold sphere, ; APL xtalk, for water/atm removal is ; the correction used for all level data ; 1 = "Placeholder" for other user-defiend correction ; VERBOSE - toggle detailed summary of actions, including output of ; hardcoded variable such as motor voltage. ; ; OUTPUTS (through nis_proc) ; Output files - several MET coded FITS files may result from processing ; xxx.000 = Uncalibrated FITS file reformatted to Calibrated FITS file, ; but not changed in any other way. ; xxx.dko = dark only file - dark spectra separated and saved. ; Overwritten each time any ; given sequence is run through NISCAL. ; xxx.rfd = uncalibrated spectra file with Spectra divided by exposure ; time, (DN, floating point). ; xxx.dks = spectra with darks subtracted, (DN). ; xxx.cdn = Calibrated DN (corrected for temp, xtalk, mirror ; position). If this is a cal target observation ; this includes the photometric correction. ; xxx.rad = Radiance. takes .cdn data and applies DN to radiance ; conversion ; xxx.iof = Converts radiance to I over F (Reflectance or Albedo). ; ; USES ; strip ; nis_proc, which calls a number of other procedures ; ; HISTORY ; Created: 12/19/97 by N.R.Izenberg (Version 1.1) ; Modified ; 07/08/98 (NRI) v2.0 Major update to all sub processes ; nis_proc command line shown as output ; 08/18/98 (NRI) Added "Science Calibration" and "The Works" options ; 10/22/98 (NRI/JB) v3.0 Complete overhaul to .conf file format and list ; processing capability ; 11/19/98 (NRI) v3.1 Add auto-output option to /project/near/ST/NIS/products ; Add new help text ; 01/11/99 (NRI) Add Dark-subtrraction Skip. ; 03/30/99 (NRI) Add Dark_mode 4 for standard dark subtraction. ; 04/06/99 (NRI) Minor Change conf_file to cfile ; 04/07/99 (NRI) Eliminate Ch 47 correction from nis_c_cross because ; Ch 47 is bad and correction is very small ; 10/14/99 (NRI) v3.6 Add "The works" calb_mode 5 does all 4 cals with one call. ; 02/06/00 (NRI) v3.7 Change in method of interleaved dark subtraction from ; sequence-based to MET-based. This should ; alleviate problems when missing data throws ; off the commanded interleave step ; Minor update to fist formatting program. ; Skip all sequence 1 calls beyond .raw ; 02/14/00 (NRI) v3.8 Fix High DN value rollover problem ; Fix to gold sphere water band correction dn2rad file. ; Change .raw to .rfd ; 02/17/00 (NRI) v3.9 Kill setting negative data to zero. in the nis_c_dark* ; procedures. This interfered with dark analysis by ; artificially removing variability in the data ; Add in clean option and compatability with nis_recal. ; 02/21/00 (NRI) v3.95 Add radiative transfer model option. ; 02/28/00 (NRI) v4.0 Add APL Xtlak radiative transfer model option. ; Add empirical "make lab sample rocks look like ; lab sample rocks" correction before radiance cal. ; 05/04/00 (JFB,NRI) v4.1 ; Saturation Check at .rfd step. Keyword NIS-023 changed ; in nis_fits_format to indicate number of saturated ; channels. Flag in nis_c_write. Text output ; Fix to darksep for single dark spectra. ; Use of NAXIS keyword instead of Size() where appropriate ; throughout. ; Spline dark subtraction option. ; 06/08/00 (NRI/DW) v4.2 Improved motor volt cor, + MVC mod to spline dark ; (Dark mode 7 and 8) ; Eliminate motor_volt variable, since total NIS dataset ; uses only 15 volt setting. ; Fix single sepdark multple spectra division ; 06/19/00 (NRI) Continued work on 4.2. Better characterization ; of spacedarks noise. placeholder for heater event flag ; Motor voltage MET table correction ; Run FY through end using 4.2 correction for Jimbo to ; make emirical correction. Incororate Dennis' heating ; event flag at that time or as intermediate 4.3 step. ; 06/27/00 (NRI/DW/JFB) v5.0 MET tables for Empirical correction. ; METfind subproc for all MET-based data. ; Heater event flag. ; 05/XX/01 (NRI) Actual 5.0 modifications for archiving. ; MOD of CNISF and all subroutines ; MOD of CONFIG file format ; -remove spicekill. ; -CHANGE rad_mode options to *2* ; rad_mode 0 is dn2rad_apl_xtalk_rtmcor.dat ; the corr used for all 5.0 data ; rad_mode 1 is "Placeholder" for other ; user-defiend correction ; -remove slit_mode. Only one slit mode is ; used throughout the in-space mission ; 07/12/01 (NRI) 5.01 tweaks include separatin gmetfind into 2 procedures ; ; NOTES ;============================================================================= ; Main Program if n_elements(help) ne 0 then begin print,"Usage is: niscal, SOURCEDIR=dir, cfile=file, /help" print," sourcedir - contains program directory information" print," Enter manually if the sourcedir environment variable" print," is not in your .cshrc" print," cfile - configuration file. Usually 'xxx.conf'. If not" print," entered manually, a pickfile widget will appear to" print," aid selection" print," cfile looks like this:" print," ## This is a NISCAL configuration file: 30 June 2001 version print," ## Please modify the parameters in the first column only, even print," ## if the entry should be blank print," /dir/dir/dir/dir/ ###DIRECTORY_OF_RAW_NIS_INPUT_DATA" print," /dir/dir/dir/dir/ ###DIRECTORY_FOR_CALIBRATED_NIS_OUTPUT_DATA" print," /dir/dir/list.data.test ###LIST_OF_RAW_NIS_FILENAMES (w. directory)" print," /dir/dir/list.dark.test ###LIST_OF_RAW_NIS_DARKS (w. directory)" print," 4 ###NISCAL1=CalibrationMode(0-5)" print," 1 ###NISCAL2=DarkMode(0-8)" print," 0 ###NISCAL3=NoiseMode(0-1)" print," 1 ###NISCAL4=RadMode(0-1)" print," 1 ###NISCAL5=VERBOSE(0-1)" print," See niscal's MANUAL.txt or comments withinin niscal.pro" print," for further details" print," NOTE - if the Output directory is denotes as 'ST' then" print," calibrated ouput will be placed in:" print," /project/near/ST/NIS/products/XY/ " print," where X is the mission phase and Y is the subphase," print," taken from the filename" print," Useful to team members running on APL computers. print," /help - displays this help message" goto, helpout endif version_id=5.01 print print, ' ##########################' print, ' # #' print, ' # NISCAL Version 5.01 #' print, ' # #' print, ' ##########################' print if not (keyword_set (SOURCEDIR)) then sourcedir=getenv('NIS_CALB') ; Source directory is where the nis_cal folder and its ; subfolders are located ; look for source directory environment variable if source ; directory is not defined if not (keyword_set (cfile)) then begin cfile=dialog_pickfile(PATH=sourcedir, $ TITLE='Choose a NISCAL configuration file', $ FILTER='*.conf') endif if (cfile eq '') then cfile=sourcedir+'niscal.conf' print,cfile modes=bytarr(5) ;; read the niscal configuration file using readcol ;; read through first for the four ascii configuration entries ;;------------------------------------------------------------ fmt='A,A' readcol,cfile,FORMAT=fmt,words,comments,SKIPLINE=3 ;help,words ;print,words ;; read through next for the six integer configuration entries ;;-------------------------------------------------------------- readcol,cfile,FORMAT='I,A',modes,comments,SKIPLINE=3 INDIR=words(0) OUTDIR=words(1) & dir = OUTDIR NIS_DATA_FILES=words(2) NIS_DARK_FILES=words(3) if n_elements(clean) ne 0 then begin print," Cleaning out old calibration files in "+outdir nis_clean, outdir, /all print endif ;initialize strings ;------------------ filelist=NIS_DATA_FILES darklist=NIS_DARK_FILES calb_mode = modes(0) dark_mode = modes(1) noise_mode = modes(2) rad_mode = modes(3) verbose = modes(4) print Print, "Configuration file inputs:" print print," Filelist = "+filelist print," Darklist = "+darklist print," Calb mode = ",calb_mode print," Dark mode = ",dark_mode print," Noise mode = ",noise_mode print," Rad mode = ",rad_mode print," Input dir = "+indir print," Output dir = "+dir print," Source dir = "+sourcedir print," Verbose = ",verbose print ;; now read DATA input list into string vectors... ;;-------------------------------------------------- readcol,nis_data_files,FORMAT='A',input_data_files ;; Now convert name list into MET lists ;;------------------------------------------ datasz=size(input_data_files) ;; Work "THE WORKS" (Calb_mode 5) to cycle through all 5 cal modes ;; if calb_mode is not 5, the the loop will only run once for the ;; desired cal level. ;;--------------------------------------------------------------- if calb_mode eq 5 then local = 0 else local = calb_mode if calb_mode eq 5 then hical = 4 else hical = calb_mode for calloop = local,hical do begin c_mode=calloop ;; Next start the loop that will run the list through nis_proc ;;------------------------------------------------------------ for k=0,datasz(1)-1 do begin ; select the files to be run through nis_proc ;-------------------------------------------- filename= input_data_files(k) ;; Find the bounding darks if the dark_mode is 1 or 2 ;;--------------------------------------------------- d1='' & d2='' if dark_mode eq 1 or dark_mode eq 2 then begin datamets=fltarr(datasz(1)) ;; Read the DARK input list into string vectors... ;; and convert name list into MET lists ;;-------------------------------------------------- readcol,nis_dark_files,FORMAT='A',input_dark_files darksz=size(input_dark_files) for i = 0,datasz(1)-1 do begin datamets(i)=double(strmid(input_data_files(i),3,9)) endfor darkmets=fltarr(darksz(1)) for j = 0,darksz(1)-1 do begin darkmets(j)=double(strmid(input_dark_files(j),3,9)) endfor dataphase=strarr(datasz(1)) for l = 0,datasz(1)-1 do begin dataphase(l)=strmid(input_data_files(l),0,2) endfor ; preceeding dark file prec = abs((darkmets * (darkmets lt datamets(k))) - datamets(k)) if min(prec) gt 1.e4 then begin print print,input_data_files(k) print,"WARNING: There is no preceeding dark file within" print,'1000 s. Using the first dark in list' print d1=input_dark_files(0) endif else begin prec2 =total(findgen(darksz(1)) * (prec eq min(prec))) d1=input_dark_files(prec2) endelse ; follwing dark file post = abs((darkmets * (darkmets gt datamets(k))) - datamets(k)) if min(post) gt 1.e5 then begin print print,input_data_files(k) print,"WARNING: There is no following dark file within" print,'10000 s. Using last dark in list' print d2=input_dark_files(darksz(1)-1) endif else begin post2=total(findgen(darksz(1)) * (post eq min(post))) d2=input_dark_files(post2) endelse endif if verbose eq 1 then begin print print,"NIS_PROC: "+filename+", if dark_mode eq 1 or dark_mode eq 2 then begin print," Dark1 = "+d1 print," Dark2 = "+d2 print endif endif If outdir eq "ST" then begin ;convert first 2 letters of filename to directory dir='/project/near/ST/NIS/products/' $ +strmid(dataphase(k),0,1)+'/'+dataphase(k)+'/' endif ; Skip sequence 1 files (these are single second snaps used to ; separate real data sequeces ;------------------------------------------------------------- if strmid(filename,13,2) eq '01' then begin if verbose eq 1 then begin print,"******* Sequence 1 Calibration Skipped *******" endif goto,seq1skip endif nis_proc, c_mode, dark_mode, noise_mode, rad_mode, $ indir, filename, version_id, verbose, $ dir, dark1=d1, dark2=d2, sourcedir=sourcedir seq1skip: If outdir eq "ST" then begin Print,'Cleaning ...' nis_clean,dir endif endfor endfor if outdir ne "ST" then begin Print,'Cleaning up...' nis_clean, outdir endif helpout: end