This is the imclean package Version 2.1.  It contains the following files:

readme.txt     this file
imclean.pro    IDL procedure, identifies bad pixels
               and calls imfixrow.pro to repair them
imfclean.pro   IDL procedure, wrapper for imclean.pro
               reads in a FITS file, cleans it, writes out a FITS file
imfixrow.pro   IDL procedure, called by imclean.pro (formerly imbadfix.pro)
               repairs pixels identified as bad, updates mask
imfixsim.pro   can be called by imclean.pro (formerly imnan.pro)
               simplified scheme to identify and repair bad pixels
idrogue.pro    IDL procedure to compare two hi-res images to generate a
               rogue pixel mask  (included mostly for backwards 
               compatibility and not recommended)
b0_omask.fits  order map for SL
b1_omask.fits  order map for SH
b2_omask.fits  order map for LL
b3_omask.fits  order map for LH

This version differs from Version 2.0 in only one way:
Some routines have been renamed in order to avoid collisions with
routines in irsclean, the improved version of this software
distributed by the SSC.

This file is organized as follows:

1.  Installation
2.  Overview of procedures
3.  Typical calls
4.  Future plans
5.  Commands and syntax
6.  Notes on bad pixel masks

1.  Installation

Unpack the tar file in a directory which you include in your IDL path.
Place the omask FITS files in a convenient location, and edit line
40 of imfclean.pro (starts with "path=") to point to the correct path 
to these files.

Make sure that you are running the current version of imclean by
typing, at the IDL prompt:

IDL>  imfclean, ... ,/version

The imfclean procedure will print that it's running version 2.1 dated 
5 Dec, 2005.  It will crash if you don't specify the right number 
of inputs, but before it does, it will print the version number.
Some users have discovered that they are running an outdated version
of imclean because they have older versions buried in their IDL path
structure.

2.  Overview of procedures

The procedure imfclean.pro operates on FITS files and serves as a
wrapper which calls imclean.pro.  It basically reads FITS files,
including the order masks, calls imclean and passes it keywords,
and writes the result to disk as a FITS file.

The function imclean.pro operates on images and mask arrays which
have already been read into IDL.  It calls various functions to 
produce a final mask of bad pixels and repair them.  It returns 
the repaired image, and if requested, an updated mask.  The 
standard use is to pass the mask associated with the image (the
bmask), ask the routine to flag NaNed data (with the /nan keyword),
and to call imfixrow.pro to repair an image, but the 
simpler function imfixsimp.pro can also be called (by invoking the
/simple keyword).  It is possible to generate a mask of rogue pixels 
by passing a rogue comparison image, which forces a call to 
imrogue.pro, but this will only work for SH and LH and is no longer 
recommended.  It is better to deal with rogue pixels by passing a 
campaign mask using the cmask parameter.

Rogue pixels are defined to be those pixels which misbehave enough
to warrant correction, but do not misbehave enough to be NaNed or
flagged in the bmask.

The function imfixrow.pro repairs one order and one row at a time.  
Flagged pixels in each row are repaired using the function 
pixfixrow.pro, which is contained in the same file.  The algorithm 
is to assemble an average profile from rows above and below the 
affected row, normalize it to the affected row using unflagged 
pixels in that row, and then replace the flagged pixels, one per 
call to pixfixrow.pro.  This algorithm is fairly robust, but the 
user should keep in mind that it will not work properly if the 
spatial profile of the spectrum changes radically in the vicinity
of a bad pixel, a situation which will be encountered, for example,
when a continuum point source is embedded within an extended
forbidden line source.

The function imfixsimp.pro constructs an average replacement value
for a pixel from pixels in the same spatial position two wavelength
elements above and below it.  This function will not work well in
a source with strong spectral variations over a short wavelength
range (such as a strong line source).  

3.  Typical calls

Here is a typical call to imfclean.pro:

IDL> imfclean,'target.1200.im.fits','target.1200.ic.fits',3,cmask='rogue_10_LH.fits',bmask='target.1200.bm.fits',outmask='target.1200.bc.fits',maskval=4096,/nan

The length is a bit awkward, but this is driven mostly by the size of the
FITS file names.  
  input   = target.1200.im.fits - the raw image
  output  = target.1200.ic.fits - the cleaned image
  module  = 3                   - for LH
  cmask   = rogue_10_LH.fits    - the rogue mask for LH, in this case 
                                  version 10 (for Campaign 10)
  bmask   = target.1200.bm.fits - the bmask image for EXPID 12, DCE 00
  outmask = target.1200.bc.fits - the cleaned bmask image
  maskval = 4096                - all pixels with a bit of 12 or higher 
                                  set will be identified and repaired
  /nan                          - also repairing NaNs

This call identifies and repairs NaNs, bmask values above bit 12, and 
rogue pixels as defined in the campaign mask.

Users are encouraged to use the /nan keyword in all calls, because we 
have discovered that pixels can be NaNed in an image without 
corresponding flags in the bmask.  Users are also encouraged to use the 
outmask parameter to save the updated bmask, because typical routines 
like SMART and SPICE will not use a corrected pixel unless it is no 
longer flagged as bad.

There are many degrees of freedom here, but two variations are common.
Use of the /simple keyword will call imfixsimp.pro instead of imfixrow.pro,
which might be wise if the spatial source geometry is a strong function
of wavelength.  Users can also convert maskval from a simple limit to
a mask to be logically and'ed with the bmask by using the /maskand
keyword.  

4.  Future plans

We have handed the software to the SSC with a new wrapper called
irsclean.pro, which they will soon distribute along with some of the
replacement functions like imfixrow.pro.  The generation of rogue
pixel masks (as done using the rogue image parameter) will be
separated.

We will continue to upgrade the imclean functions as well.  We are 
planning to implement improvements to imfixsimp.pro which make it
more robust in future versions.  These are being developed at
the University of Rochester.  

Users are encouraged to write their own wrappers as well.

5.  Commands and syntax

This section provides a more detailed description of the functions
and how they should be used.

imclean   Operates on images after they have been read from disk
          Identifies bad pixels, then calls imfixrow.pro or 
          imfixsimp.pro to repair them.

          Syntax:  clean_image=imclean(input_image,omask_image,KEYWORDS)

          input_image - the image to be cleaned
          omask_image - the order mask for the relevant IRS module
                        This will normally be loaded and passed by imfclean

          The user will need to set at least one of the following keywords:
          /nan               - To identify and replace NaNs
          maskim=bmask_image - To pass the bmask image
          rogue=rogue_image  - To identify and replace rogue pixels
                               To be used only with SH and LH images
          /simple            - To use a simpler algorithm to identify and
                               correct NaNs using imfixsimp.pro

          maskim option:
          The  user can set either maskval or use the default maskval of 16384
          All pixels with a bmask value equal to or greater than maskval are
            flagged as bad 
          /maskand - optional keyword to AND the maskval with the bmask values
             This is a bit more sophisticated than the default which is a 
	     simple limit.

          rogue option - use this to generate a rogue mask from a comparison
	      image.  Remember that this option only works for SH and LH, and
	      is no longer recommended, now that rogue masks are available.
            This calls imrogue.pro, which identifies rogue pixels by using the
              ratio of the two nods (after shifting the images to line the nods
              up).
          /flip - use this keyword if operating on Nod 2 data in LH or Nod 1
	          data in SH
          threshold=threshold - use this to change the identification threshold
            from the default value of 10.  A higher threshold reduces the
            number of pixels identified as rogue pixels.
          When invoking the rogue option, the user passes a rogue comparison
            image, which is the corresponding image in the other nod position.
            If cleaning a Nod 1 image, then pass Nod 2 as the rogue image
	      Use the /flip keyword for SH data.
            If cleaning a Nod 1 image, then pass Nod 1 as the rogue image AND
              set the /flip keyword for LH data (but not SH).
            The /flip option tells imrogue.pro which way to shift the images
              so that the nods lie on top of each other.
      
         Other options

         maskout=output_mask_image 
           STRONGLY ENCOURAGED - invoked in imfclean.pro by setting outmask
           This output_mask should be used in place of the bmask file when
             extracting spectra using SMART, SPICE, or the offline pipeline.
           Otherwise, the extraction software will assume that the pixel you
             have just replaced is still bad, leaving a divot in your spectrum.
           NOTE:  corrected pixels are reset to bmask=0, which is simplistic.
           ANOTHER NOTE:  This option is invoked from imfclean with "outmask"

         order=order to limit bad pixel replacement to one order.  Passed
           to imfixrow.pro
         nrow=number_of_rows used to determine the replacement value for the
            bad pixel.  The default is 3 rows, to either side of the bad pixel.
            This parameter is used within pixfixrow, which is in the file
            imfixrow.pro.  The default should work fine.
	 limit=threshold for a pixel to be used to normalize the replacement
	    profile to row with the bad pixel.  The default is 0.025 times the
	    total of the replacement profile.  This limit is passed to
	    pixfixrow, and it prevents it from fitting the profile to noise.
         /verbose - to turn on diagnostic output within pixfixrow 
	    (in imfixrow.pro)
         /zero - option to zero out-of-order data.  Not recommended except for
           diagnostics.
         niter=number_of_iterations for imfixsimp.pro
           Ignored unless /simple is set, default value is 5

imfclean  Wrapper for imclean, operates on FITS files.

          Syntax:  imfclean,in_file,out_file,module,KEYWORDS

          infile - the FITS file containing the image to be cleaned
          outfile - the cleaned FITS file
          module - 0,1,2,3 for SL,SH,LL,LH
            Tells imfclean which order mask to read in.

          All of the keywords listed in the imclean section will be passed to 
          imclean.pro and subordinate procedures

          Other options:

          bmask=bmask_FITS_file - loaded and passed to imclean as the bmask.
          rogue=rogue_FITS_file - loaded and passed to imclean as the 
	                          comparison image to generate a rogue mask.
				  Not recommended.
          outmask=new_FITS_file - STRONGLY ENCOURAGED
            All replaced pixels will have the mask value reset to zero
            Identifying the outmask FITS file name will let imfclean.pro
              save this new mask so that it can be used by the extraction
              software
            See the notes for maskout in the imclean section above.
          version               - use to check which version you are running

6.  Notes on bad pixel masks

We have learned that simply correcting NaNed pixels and pixels flagged
in the bmask files is insufficient.  To produce higher quality spectra
from the images, it is also necessary to correct the rogue pixels.
These pixels have been damaged by radiation, but not to a degree
sufficient for them to be flagged in the bmask file or NaNed in the
images.  They still misbehave, and it is not difficult to identify
some of them visually when examining two-dimensional images.

The SSC has developed a techniques to identify rogue pixels more
carefully and provides masks of the rogue pixels for each campaign
at:  http://ssc.spitzer.caltech.edu/irs/roguepixels/

The simplest means to mitigate for rogue pixels is to subtract a
background image from the on-source image before extracting spectra,
but these are not always available, and for some of the pixels, a
simple background subtraction is not really sufficient.  Using the
rogue masks (with the cmask keyword in calls to imfclean.pro)

There is an ongoing debate within the IRS Science Team as to whether
it is advantageous to apply what has been referred to a grand rogue
mask, which is a map of any pixel which has been identified as a 
rogue, even in only one campaign.  This single grand rogue mask can
then be applied to all campaigns.  The motivation is that pixels
can misbehave in a given campaign without being identified as rogues
in that particular campaign, but the odds are good that have been
identified at one time or another.  This approach is certainly 
safest, but it runs into the danger of producing so many pixels 
flagged as bad that the correction algorithm can break down.  In
addition, many of the pixels identified as rogues in only one or
two campaigns do behave properly most of the time.  Users are 
encouraged to download the rogue pixel information from
the SSC and experiment with masking techniques.

This readme.txt file is dated 8 August, 2005 and corresponds to 
the imclean package version 2.0.
