Manual for reconstructing tomographic images with PyHST

a description and the lastest sources for ESRF PyHST project can be found here

a list of useful Unix commands can be found here


Table Of Content

basics required:
keep in mind:
  • mostly all programs can be found via /hmi/bin (nitomo machines) or /chmi/nitomo1/bin (any other HMI intranet unix machine) and corresponding subfolders (/hmi/bin/BAM - BAM programs, /hmi/bin/ESRF - ESRF programs etc.), nearly all contain a help menu which can be accessed by running the program without any parameters (eg /hmi/bin/bam2hst), on Linux 64bit-machines (nitomo3) old 32-bit binaries can be found always via /hmi/bin32 while 64bit-applications are located in /hmi/bin (32bit binaries work on 64bit systems but not vice versa)
  • PyHST is available for Linux only (nitomo1, ...), hst_master, hst_slave are available for Tru64 (darling0, nacha ...) only; additional Tru64 software can be recognized by the *_alpha suffix, binaries without this suffix are Linux-only
  • graphic display is important - only use ssh -X account@domain.whatever, Windows user should enable "Tunnel X11 connections" in their SSH option menu and make sure to start X-Win before launching the ssh application; test graphic output eg with "gv" - if it fails check the display parameter echo $DISPLAY and correct it with setenv DISPLAY your_host:display, for more information see DV pages
  • Unix is case sensitive - "PyHST my_file.par" is NOT the same as "pyhst my_FILE.par"
  • DO NOT USE blank space or hyphen in Unix file names (try "my_file" instead of "my file" / "my-file")
  • all data is stored LITTLE ENDIAN (LowByteFirst) - BIG ENDIAN (HighByteFirst) can lead to unexpected errors
  • check results after each step by creating images (corrected radiographics, sinograms, sample slices) and store them in easy-to-use format (JPG, TIF, etc)
  • when you are finished with the reconstruction do not forget to remove your EDF files to lower hard disc consumption - not-necessary over-quota use of hard disc space can lead to random data removement by your system administrator

All data located outside your $HOME is not backuped automatically - users are self-responsible for their backups, please ask your local contact!

1. Short notes on the EDF format (if already known skip to next step)

a)

(i) DB files (only dark correction applied) recorded with BAM-Linux-Software "ct"
(mostly all data recorded during and after June 2004)

assuming radiographic images have been recorded at BESSYII BAMline with the 2048x2048 Princeston CCD camera (2048x2048 CCD chip), your files (flat*.db and sample*.db) preferable should be located now at /hmi/Messdaten/BESSY_session_date/RAW and saved in DB format - BAM format to store image data, only the dark correction has been applied (probably during the measurement)

the raw radiographic image data has to be corrected (flatfield correction) and re-arranged (x/y-shifts to avoid ring artefacts) which is done by the program corrct, so enter the directory with your files, eg on nitomo1

cd /hmi/Messdaten/BESSY_session_date/RAW

and run

/hmi/bin/BAM/corrct -l -i=your_file.db -o=../DB/your_outputfile.db -f=your_flatfield_file.db -X -Y

parameters:

(ii) SPE files recorded with "Roper Scientific WinView32"
(mostly all data recorded before June 2004)

assuming radiographic images have been recorded at BESSYII BAMline with one of the two Princeston CCD cameras (1024x1024 or 2048x2048 chip), your files (dark*.SPE, light*.SPE, sample*.SPE and corresponding headerfiles) preferable should be located now at /hmi/Messdaten/BESSY_session_date/SPE and saved in SPE format - delivered by the camera's own software "Roper Scientific WinView32"

the SPE files have to be converted into the BAM-format DB (which basically means flat field and dark correction and data rearrangement like swab byteorder if necessary etc., values will be saved as unsigned integer 2byte) and preferable should be stored in /hmi/Messdaten/BESSY_session_date/DB - please keep in mind that the following steps are based on using these paths, so enter the directory with your files, eg on nitomo1

cd /hmi/Messdaten/BESSY_session_date/SPE

the BAM programs to use are spe2cthdn8 (1024x1024 CCD camera) or spe2ctnc3 (2048x2048 CCD camera) and should be called like this

/hmi/bin/BAM/spe2ctnc3 -h=your_headerfile -o=../DB/your_outputfile.db -l -m -X

parameters:

  • -h=your_headerfile - specifies which headerfile to use, this file is in textmode and can be edited with vi, emacs etc, it gives the path to your SPE file with the radiographs and light recordings as well as the path to your dark_*SPE file and further information which is used for the BAM header
  • -o=../DB/your_outputfile.db - specifies where the output datafile should be saved
  • -l - use LittleEndian (LowByteFirst) byteorder in *DB output file (necessary!)
  • -m - perform Median filter on data set (recommended!)
  • -X - toggles the orientation for correcting the X-axis, the only way to determine if this parameter has to be set or not is try_and_error: convert the file, display the sinogram (see below) if it is not ok, try again without -X
    having a look at the sinogram(s) is an extremely useful step to do before any reconstruction - various errors can be avoided
  • experienced users can get a full list of parameters by entering /hmi/bin/BAM/spe2ctnc3 --help

new parameters

  • BICUBIC=int constant

    interpolates more projections, for eg constant=10 6000 instead 600 projections where jused by interpolating between the experimental data (Paul Tafforeau)

  • SUMRULE={0, 1, 2}

    • 0 = use classical RAM-LAK Filter, frequency f0 is set to 0, resulting in mean(slice) as offset substracted from all attenuation values, recommended if float volumes are reduced to char-volumes after reconsturction
    • 1 = adds mean(sinogram)-mean(slice) to all attenuation values, works fine if object does not leave field of view during tomographic scan, leads to more exact determination of the local attenuation coefficient
    • 2 = impuric approach for local tomography (object leaves field of view during scan) in order to have same effect as option "1", approach: use mean of local frequencies around f0 (Mirone)

  • OUTPUT_SINOGRAMS={YES, NO}

    creates Sinograms (EDF-Format) instead of reconstructing

  • RECONSRUCT_FROM_SINOGRAMS={YES, NO}

    works now fine with PyHST and hst_slave

  • OPTIONS= { 'padding':'E' } or { 'padding':'0' }

    • { 'padding':'E' } - padds in frequency domain with last and first values of array (default!) in the sense of [FFT_of_input_data_DIMx][PADDING_FirstValue_DIMx][PADDING_LastValue_DIMx]
    • { 'padding':'0' } - padds in frequency domain with zeros

  • Fourier-Filters/Windowing, if you prefer alternative filters de-comment one of the following
        # HANN-filter
        #def FOURIER_FILTER(self, x): return 0.5 + 0.5*math.cos(2*math.pi*x)
    
        # BLACKMAN-filter
        #def FOURIER_FILTER(self, x): return 0.42 - 0.5*math.cos(2*math.pi*(x-0.5)) + 0.08*math.cos(4*math.pi*(x-0.5))
    
        # HAMMING-filter
        #def FOURIER_FILTER(self, x): return 0.54-0.46*math.cos(2*math.pi*(x-0.5))
    
        # COSINE-filter
        #def FOURIER_FILTER(self, x): return math.cos(math.pi*(x))
    
        # SHEPP-LOGAN-filter
        #def FOURIER_FILTER(self, x): return math.sin(math.pi*(x))/(math.pi*(x))
    
        # BARTLETT-filter
        #def FOURIER_FILTER(self, x): return 1 - abs(2*x)
        

    for further descriptions please refer to

    • Principles of Computerized Tomographic Imaging, by Avinash C. Kak, Malcolm Slaney, Society for Industrial & Applied Mathematics, U.S. (re-issue, 2001)
    • Reconstructions from Radiographs and Electron Micrographs, G. N. Ramachandran, A. V. Lakshminarayan, Proc. Acad. Sci. USA 68, 2236 (1971)
    • Einführung in die Computertomographie Mathematisch-physikalische Grundlagen der Bildrekonstruktion, by T. M. Buzug, Springer Verlag (2004).

  • Ring-artefacts type I (warm CCD pixel, defects on scintillating screen etc.), filters

    • sinogram-based filter, performs high-pass on mean over all angles (of one slice) to detect distortions, these are than corrected by adding the high-passed sinogram to input sinogram with conservation of the mean of the input sinogram, the array ar[] as heaviside-function defines the high-pass
            DO_SINO_FILTER= YES
            SINO_FILTER = "SINO_Filter"
            ar = Numeric.ones(NUM_IMAGE_1,'f')
            ar[0]=0.0
            ar[2:18]=0.0
            SINO_FILTER_PARA={"FILTER": ar }
            
      the path to the object (in case of conflicting versions, selfdone objects etc) can be set via
            SINO_FILTER_PATH="/"
            

      initially suggest by Mark River

    • same as above, only median instead of mean is used
            DO_SINO_FILTER= YES
            SINO_FILTER = "SINO_Filter_Median"
            ar = Numeric.ones(NUM_IMAGE_1,'f')
            ar[0]=0.0
            ar[2:18]=0.0
            SINO_FILTER_PARA={"FILTER": ar }
            
      the path to the object (in case of conflicting versions, selfdone objects etc) can be set via
            SINO_FILTER_PATH="/"
            
    • another filter based on detecting lines parallel to the angle axis can be called via
            DO_SINO_FILTER= YES
            SINO_FILTER = "SINO_Filter_Smooth"
            span = 51
            SINO_FILTER_PARA={"SmoothFilter":span}
            
      the detected lines are removed by a "Moving Average Filter", for further details see manual (Mirko Boin)
    • to be done: sinogramic filter based on filtering the 2D FFT sinogram, see C. Raven, Numerical removal of ring artifacts in microtomography, Rev. Sci. Instr. 69, no. 8 (1998)

    • Boin filter ...

  • Ring-artefacts type II (inadequateness of flatfield correction), Tafforeau filter on projections, to be done

  • Axis correction, in case of axis movement during tomographic scan (inadequateness of mechanics)
        DO_AXIS_CORRECTION=YES
        AXIS_CORRECTION_FILE="your_file"
        
    your_file has to be text-formated, a column of int values (number of rows == number of projections) defines a shift (pixel units) in x-direction for each projection used, a second column can be used to introduce corrections in the y-direction as well. The latter case includes shifting based on interpolating while only x-corrections are exactly performed. X-corrections are useful to realign mismatches after sample has been moved out and in beam for flatfield projections.

  • Median-Filter

    • relative conditioned, image-median(image) is used to determine wether filter should be applied or not
          DO_CCD_FILTER= YES
          CCD_FILTER = "CCD_Filter"
          CCD_FILTER_PARA={"threshold": 0.0005 } #filter condition
          

      Via the threshold parameter one can control how strong the filter works on the image, entering -1.0 means max filtering. The path to the compiled object can be set via

          CCD_FILTER_PATH="/"
          
      in case you are running into conflicting versions or want to try on your own. To be done: Parameter for chosing filter application for or after flatfield correction or both
    • absolute conditioned, same as above with new parameters FRAC and ABS to be used as

          DO_CCD_FILTER= YES
          CCD_FILTER = "CCD_FilterAbsRel"
          CCD_FILTER_PARA={"threshold": 0.0005, "FRAC": 1, "ABS": 1}
          
      with the path can be specified via
          CCD_FILTER_PATH="/"
          
      ABS=1 will use abs(median(image)) instead of median(image) while FRAC=1 toggles between comparing the threshold with a) abs(median(image))/abs(image) (case ABS=1) or b) median(image)/abs(image) (case ABS=0) and plain comparison of abs(median(image)) or median(image) with the given threshold

    • CCD_Filter_Min
      is a test plugin not to be used at all

how to use forward projection

input: volumes volume, output: projections

b) here we assume that however you have one or more DB files which contain your data to reconstruct into one tompgraphic image (no matter which program has been used for aquiring the images)

if necessary the BAM header of your *DB file can be viewed and edited with

/hmi/bin/BAM/header your_file_in_BAM_format

or alternatively can be display with XDISP, enter the directory where the tomographic results should be stored on nitomo1, eg

cd /hmi/username/BESSY_session_date/sample_XYZ/

and create a folder to contain the radiographic images and enter it

mkdir radios

cd radios

the program to use is bam2hst and should be called like this

/hmi/bin/bam2hst your_db_file

the program will display the necessary BAM header information and write all radiographic images included in the DB file to single (one file - one image) EDF files (EDF format - 1024 byte text header plus binary data), the logarithm will be applied and the image values are stored as FLOAT values (4 byte)

options:

  • if your images are stored in more than one DB file it is necessary to call bam2hst more than once, in that case one has to re-initialise the file_index_counter otherwise all previously created EDF files will be overwritten, use
    /hmi/bin/bam2hst your_db_file -p [number_of_last_image_already_created+1]
  • it can happen that the axis of rotation is pertubated (eg hang up during data capture) from one DB file to the next one (look for discontinuities in the sinograms), that error sometimes can be corrected by using the -shift [INT] option (only if the shift is constant for one DB file and only in radiographic mode)
  • in order to create sinograms instead of radiographic images use
    /hmi/bin/bam2hst -sino your_db_file (in order to create a sinogram out of a set of EDF files use /hmi/bin/sinomaker in the directory where the EDF files are stored)
  • experienced users can get a full list of parameters by entering /hmi/bin/bam2hst

some nice tools for pre-processing your corrected DB files

are located in the program ct2ct which is used like this

/hmi/bin/BAM/ct2ct -l -i=your_input_file.db -o=your_output_file.db

options:

  • -l - use LittleEndian (LowByteFirst) byteorder in *DB output file (necessary!)
  • -m - perform Median filter on data set (recommended, run Median twice - before (via ct2ct) and after log() (PyHST) - to erase heavy noise but keep in mind that you are losing resolution as well)
  • -n=X -N=Y - normalises frames with an average of 10 x 10 pixel area window (has always to be without sample = only background information) starting at X (in pixel units) and Y (in pixel units)
  • --help - more options for the experienced user
Sample_Noisy Sample_DeNoise_2xMedian
noisy image
denoised by applying median twice


How to display the data:

2. Creating HST parameter file (if already done skip to next step)

paramter files for HST can be created with hst_master which is a Tru64 program, to run it log on an alpha machine, eg from nitomo1

ssh -X your_account@darling

and run

/chmi/nitomo1/bin/ESRF/hst_master_new

accept the "Conditions Of Use" and then select "Reconstruction Direct From Sinograms: NO" (reconstruction from sinograms at the time this manual is written is only possible with HST_SLAVE!). Then you are asked for the first data file image, use the browser to find your directory (eg /chmi/nitomo1/username/BESSY_session_date/sample_XYZ/radios) and select the first image in that folder (should be radio0000.edf) by plain clicking. The following screen appears (color palette can be different)

Select a region of interest (ROI) - first click activates, second click finalises - by drawing a rectangle. This ROI is used to crop the reconstruction area, the smaller the ROI the faster the reconstruction and less hard disc space is required. The next screen you have to select the last data file image by plain clicking and then onwards into the parameters, the first mask should look like this

and on the next one

the following parameters should be checked, the rest should be left unchanged,

After passing the parameters once again you have to select a ROI (see above) and afterwards the "180 degree rotation image file" (usually again the last image in your folder), if the selected image is displayed correctly click on YES and THEN for the last time select a region of interest (this time used to calculate the axis of rotation - principle: use center of mass from 0 degree and 180 degree image, in the middle you have the axis of rotation). Press ok on the next screen and on the last screen you can save the parameters to a file of your choice, it is recommended to use the suffix .par!

More experienced users can skip the graphical interface and create a parameter file directly by editing a file from a previous reconstruction. A parameter file for HST looks like this

! HST_SLAVE PARAMETER FILE

RECONSTRUCT_FROM_SINOGRAMS = NO

! Parameters defining the projection file series
FILE_PREFIX = /hmi/rack/Bessy_Okt03/alex_AlSi7_16wpTiH2_beh/radios/radio
NUM_FIRST_IMAGE =      0 ! No. of first projection file
NUM_LAST_IMAGE =    899 ! No. of last projection file
NUMBER_LENGTH_VARIES = NO
LENGTH_OF_NUMERICAL_PART =  4 ! No. of characters
FILE_POSTFIX = .edf
FILE_INTERVAL =      1 ! Interval between input files

! Parameters defining the projection file format
NUM_IMAGE_1 =    991 ! Number of pixels horizontally
NUM_IMAGE_2 =   1251 ! Number of pixels vertically
IMAGE_PIXEL_SIZE_1 =      3.59 ! Pixel size horizontally (microns)
IMAGE_PIXEL_SIZE_2 =      3.59 ! Pixel size vertically

! Parameters defining background treatment
SUBTRACT_BACKGROUND = NO ! No background subtraction
BACKGROUND_FILE = N.A.

! Parameters defining flat-field treatment
CORRECT_FLATFIELD = NO ! No flat-field correction
FLATFIELD_FILE = N.A.
FLATFIELD_CHANGING = N.A.
FF_PREFIX = N.A.
FF_NUM_FIRST_IMAGE = N.A.
FF_NUM_LAST_IMAGE = N.A.
FF_NUMBER_LENGTH_VARIES = N.A.
FF_LENGTH_OF_NUMERICAL_PART = N.A.
FF_POSTFIX = N.A.
FF_FILE_INTERVAL = N.A.

TAKE_LOGARITHM = NO

! Parameters defining experiment
ANGLE_BETWEEN_PROJECTIONS =    0.20000 ! Increment angle in degrees
ROTATION_VERTICAL = YES
ROTATION_AXIS_POSITION =       568 ! Position in pixels

! Parameters defining reconstruction
OUTPUT_SINOGRAMS = NO ! Output sinograms to files or not
OUTPUT_RECONSTRUCTION = YES ! Reconstruct and save or not
START_VOXEL_1 =     1  ! X-start of reconstruction volume
START_VOXEL_2 =     1 ! Y-start of reconstruction volume
START_VOXEL_3 =     210 ! Z-start of reconstruction volume
END_VOXEL_1 =    990 ! X-end of reconstruction volume
END_VOXEL_2 =    990 ! Y-end of reconstruction volume
END_VOXEL_3 =    210 ! Z-end of reconstruction volume
OVERSAMPLING_FACTOR =  4 ! 0 = Linear, 1 = Nearest pixel
ANGLE_OFFSET =  -5.000000 ! Reconstruction rotation offset angle in degrees
CACHE_KILOBYTES =   4096 ! Size of processor cache (L2) per processor (Kbytes)
SINOGRAM_MEGABYTES =    800 ! Maximum size of sinogram storage (megabytes)

! Parameters defining output file / format
OUTPUT_FILE = /hmi/rack/Bessy_Okt03/alex_AlSi7_16wpTiH2_beh/alex.vol

! Reconstruction program options
DISPLAY_GRAPHICS = NO ! No images

Section-by-Section description:

! HST_SLAVE PARAMETER FILE

RECONSTRUCT_FROM_SINOGRAMS = NO

toggles if input files are in radiographic style or as sinograms written, sinograms (at the time this manual was written) can only be used with hst_slave, not with PyHST

! Parameters defining the projection file series
FILE_PREFIX = /hmi/rack/Bessy_Okt03/alex_AlSi7_16wpTiH2_beh/radios/radio
NUM_FIRST_IMAGE =      0 ! No. of first projection file
NUM_LAST_IMAGE =    899 ! No. of last projection file
NUMBER_LENGTH_VARIES = NO
LENGTH_OF_NUMERICAL_PART =  4 ! No. of characters
FILE_POSTFIX = .edf
FILE_INTERVAL =      1 ! Interval between input files

widely self-explaining, "FILE_PREFIX" should be set to the path where you data image files are located plus "radio", the rest should be left unchanged as long as you use bam2hst to create EDF files. "FILE_INTERVAL" is necessary if you want to reconstruct quick&dirty, eg setting it to "2" makes hst recognizing only half of input files.

! Parameters defining the projection file format
NUM_IMAGE_1 =    991 ! Number of pixels horizontally
NUM_IMAGE_2 =   1251 ! Number of pixels vertically
IMAGE_PIXEL_SIZE_1 =      3.59 ! Pixel size horizontally (microns)
IMAGE_PIXEL_SIZE_2 =      3.59 ! Pixel size vertically

the first two values can be found in any of your EDF files (belonging to your current reconstruction) header - use head -c 1024 radio0000.edf to display header information and then Dim_1 = NUM_IMAGE_1 and Dim_2 = NUM_IMAGE_2, the image pixel size is the resolution used (usally something like 3.6, 5.5 or 11 microns) and can be found in your BAM headerfile, see "Scandx=..." [mm]

! Parameters defining background treatment
SUBTRACT_BACKGROUND = NO ! No background subtraction
BACKGROUND_FILE = N.A.

! Parameters defining flat-field treatment
CORRECT_FLATFIELD = NO ! No flat-field correction
FLATFIELD_FILE = N.A.
FLATFIELD_CHANGING = N.A.
FF_PREFIX = N.A.
FF_NUM_FIRST_IMAGE = N.A.
FF_NUM_LAST_IMAGE = N.A.
FF_NUMBER_LENGTH_VARIES = N.A.
FF_LENGTH_OF_NUMERICAL_PART = N.A.
FF_POSTFIX = N.A.
FF_FILE_INTERVAL = N.A.

can be left unchanged as long as BAM software as well as bam2hst is used, the flatfield and dark correction is already applied there

TAKE_LOGARITHM = NO

should be left unchanged, the logarithm is taken by bam2hst, in some cases it might be helpful to take the logarithm twice in order to improve the contrast but it can also lead to unexpectable errors

! Parameters defining experiment
ANGLE_BETWEEN_PROJECTIONS =    0.20000 ! Increment angle in degrees
ROTATION_VERTICAL = YES
ROTATION_AXIS_POSITION =       568 ! Position in pixels

"ANGLE_BETWEEN_PROJECTIONS" usually is 180 / number_of_projections and can also be found in your BAM headerfile, see "WinkelIncr=...", the "ROTATION_VERTICAL" is always "YES", the "ROTATION_AXIS_POSITION" can be determined with hst_master (see above), during the measurements, by running /hmi/bin/center_of_image -fit in the directory where your EDF files are stored and/or later (should always be) optimised by hand (see below)

! Parameters defining reconstruction
OUTPUT_SINOGRAMS = NO ! Output sinograms to files or not
OUTPUT_RECONSTRUCTION = YES ! Reconstruct and save or not
START_VOXEL_1 =     1  ! X-start of reconstruction volume
START_VOXEL_2 =     1 ! Y-start of reconstruction volume
START_VOXEL_3 =     210 ! Z-start of reconstruction volume
END_VOXEL_1 =    990 ! X-end of reconstruction volume
END_VOXEL_2 =    990 ! Y-end of reconstruction volume
END_VOXEL_3 =    210 ! Z-end of reconstruction volume
OVERSAMPLING_FACTOR =  4 ! 0 = Linear, 1 = Nearest pixel
ANGLE_OFFSET =  -5.000000 ! Reconstruction rotation offset angle in degrees
CACHE_KILOBYTES =   4096 ! Size of processor cache (L2) per processor (Kbytes)
SINOGRAM_MEGABYTES =    800 ! Maximum size of sinogram storage (megabytes)

"OUTPUT_SINOGRAMS" can be used to create sinograms instead of using bam2hst or xdisp but only works with hst_slave, not with PyHST (sinograms will be stored in EDF format (1024 byte textheader plus binary data) and can be displayed with ImageJ); "OUTPUT_RECONSTRUCTION" should always be set to "YES" (otherwise no data will be saved), "START_VOXEL_..." and "END_VOXEL_..." defining the 3D region of interest (see below), the "ANGLE_OFFSET" is explained in detail in the next step and the rest should be left unchanged

! Parameters defining output file / format
OUTPUT_FILE = /hmi/rack/Bessy_Okt03/alex_AlSi7_16wpTiH2_beh/alex.vol

the file where the reconstructed data should be written to, please use the suffix .vol !

! Reconstruction program options
DISPLAY_GRAPHICS = NO ! No images

should be set per default to no.

The latest generation of PyHST understands some more parameters which are listed here .

Eg a Median filter can be used (only with PyHST) by adding the following lines to your parameter file


#konditionaler Medianfilter
DO_CCD_FILTER= YES
CCD_FILTER = "CCD_Filter"
CCD_FILTER_PARA={"threshold": 0.0005 } #Bedingung/Schwellwert

Via the threshold parameter one can control how strong the filter works on the image, entering -1.0 means max filtering. The path to the compiled object can be set via

CCD_FILTER_PARA ={}
in case you are running into conflicting versions or want to try on your own. To be done: Parameter for chosing filte application for or after flatfield correction or both

3. Reconstruction (if already done skip to next step)

Before reconstructing a whole volume the parameter file delivered by hst_master (or self-done) has to be checked and optimised.

Please keep in mind that calculation time and resources are and always will be limited, specially on our Linux machines nitomo... which should be dedicated mostly to 3d image analysis. Therefore try to use as much as possible the HMI alpha-cluster darling (see DV pages for more details) or other machines available. Only the first node of darling is accessable via a shell (eg ssh -X darling). Single slices always can be calculated there, if the load factor on darling0 is high (check with top or uptime) it is recommended that long reconstructions are done on higher knodes using the prun program (see DV pages or enter man prun).

The fastest way for optimising is to work on a single slice only. The number of slices available is equal to the "NUM_IMAGE_2" value in your parameter file. Chose the middle slice by setting "START_VOXEL_3 = END_VOXEL_3 = NUM_IMAGE_2 / 2", the parameter file is in textformat and can be edited with (x)emacs, vi, kwrite, kate etc.

To reconstruct on nitomo1 use the following syntax

/hmi/bin/ESRF/PyHST your_parameter_file 2 FLOAT

here "2" means the number of parallel threads to be used on one machine (2 is the recommended value if you are the only one working on a dual-cpu machine, otherwise chose 1) and "FLOAT" is the arithmetic to be used (currently on Linux-PyHST only FLOAT is available, on SUN-PyHST only INT is available).

To reconstruct on darling, nacha or any other alpha-machine use

/chmi/nitomo1/bin/ESRF/hst_slave_v30 your_parameter_file

The resulting data is saved in the file specified with the "OUTPUT_FILE" parameter in raw/binary format as FLOAT 4byte values. Additionally a file *.vol.info is created where the file dimensions can be found. To view the file enter less your_file.vol.info ! (Do not try to display binary data with the less program, it might reset your terminal!)

To display the calculated slices (compareable to displaying EDF files) enter cd /hmi/ImageJ and ./run then use FILE -> IMPORT -> RAW and select the file to display, then enter the following import information:

ImageType: 32bit REAL
WIDTH: xdim_of_info_file (see below)
HEIGHT: ydim_of_info_file (see below)
Offset to First Image: 0
Number Of Images: 1  (can be higher if you calculated more than one slice)
Gap: 0
White is Zero: disabled
Little Endian: enabled
Open All Files: disabled
the x and y dimensions are the ones from the info file (NUM_X and NUM_Y values).

0) create a sinogram

The first step which has to be done before any other is to create a sinogram. Mostly all errors which can happen can be detected by checking the sinogram! So please enter the directory with your radiographic EDF files and run

/hmi/bin/sinomaker

the program widely parametrises itself! An xv is automatically opened when the sinogram has been created.

PLEASE NOTE THAT FROM NOW ON NO LONGER USER-SUPPORT IS GIVEN BY RECONSTRUCTION PROBLEMS IF YOU ARRIVE WITHOUT sinogramIC SAMPLE IMAGE IN YOUR HANDS!!!

a) optimise axis of rotation

Can be done straight-forward: hst_master delivers an axis_of_rotation value based on center of mass calculation. If you do not want to use hst_master a good estimate is always the middle of your radiogramm (NUM_IMAGE_1 / 2). A very good estimate for the axis of rotation (error usually in the range of one pixel) is calculated by running /hmi/bin/center_of_image -fit in the folder where your corresponding EDF files are stored. The latter program takes the first image (0 degree rotation) and last image (180 degree rotation), mirrors vertically the last image and calculates the L2-norm of the difference (first minus last) image. The horizontal shift of the last image is introduced as parameter - a (1,10) evolution strategy fits this shift value to the minimal L2-norm. The resulting shift value is used to calculate the axis of rotation. An uncertainness of one pixel remains due to possible rounding errors. The program widely parametrises itself! Depending on the image size and how tricky it is for the program to find the minimal value the calculation time can be up to 20 minutes.

So do the reconstruction and display the result, an example for a completely wrong axis of rotation (210 pixels away from the correct value) is displayed in the following image

You probably recognise the semi circle artefacts (180 degree scan, for 360 degree the semi circles become full circles) which are the main characterisation for a wrong axis of rotation. The aim is first to decrease the diameter of the artefacts: with big steps for huge artefacts and small steps for little artefacts as displayed in the next image

Here the axis of rotation is only 10 pixels away from the correct value, the artefacts' diameter is much smaller, if you step to far (over the minimum) by varying the axis of rotation value one can notice an inversion of the semi circles' orientation, see here

If you already come close to that point it is the best way to change the axis of rotation value only one pixel, calculate and make a whole serie around that minimum you found. Display all the slices on one screen and select the best with the eye, in this case it looks like this

Please use the same contrast setting for all images of your serie (in ImageJ use CTRL+SHIFT+C to enter the contrast menu, chose "Set" and then enter corresponding MIN_VALUE and MAX_VALUE, use the same MIN_VALUE and MAX_VALUE for each new image you have created).

b) optimise angle offset

The "ANGLE_OFFSET" has to be optimised before a best crop (min. volume) can be done! Shifting the offset to positive values rotates the sample counterclockwise, shifting it to negative values rotates clockwise. Result of optimal "ANGLE_OFFSET" is shown below.

c) crop/region of interest (ROI)

"START_VOXEL_..." and "END_VOXEL_..." defining the 3D region of interest and should be set to values which minimize the volume to be reconstructed (in order to save calculation time and hard disc space). A good tool again is ImageJ: import a slice and with the xy-coordinate display the values for optimal cropping can be calculated.

d) check and reconstruct

Make sure your region of interest is not to small (eg in cases when your sample is not parallel or not 100% vertical within the image, one can easily check that by looking at the radiographic images) - try to reconstruct the first, middle and last slice. Did you save sample slice(s)? Did you have a look at at least one sinogram? Did you save one or two sinograms and radiographs in JPG?

To reconstruct on nitomo1 use the following syntax

/hmi/bin/ESRF/PyHST your_parameter_file 2 FLOAT &

here "2" means the number of parallel threads to be used on one machine (2 is the recommended value) and "FLOAT" is the arithmetic to be used (currently on Linux-PyHST only FLOAT is available, on SUN-PyHST only INT is available). The "&" makes your program work in background which means it will continue working even your terminal dies or you when you do a logout.

To reconstruct on darling, nacha or any other alpha-machine use

/chmi/nitomo1/bin/ESRF/hst_slave_v30 your_parameter_file &

when you are finished with the reconstruction do not forget to remove your EDF files to lower hard disc consumption

e) vgstudiomax and AVS

A powerful rendering tool for displaying 3d data sets is Visual Graphics Studio - currently it is only accessable on nitomo2 (32bit) and nitomo3 (64bit). Start it by entering vgstudiomax in any directory on nitomo2/3. On nitomo2 files with a size up-to 2GB can be loaded (if your 3d image is sized in the range of 2GB ... 4GB you have to work with image stacks), one thread can use up to 4 GB. On nitomo3 the filesize and usable memory is only limited by the available system memory.

To import a (float 4byte) volume select FILE -> IMPORT -> RAW Volume and browse to your volume file which you like to display and then with NEXT to the Data Type interface:

  • 32bit FLOAT
  • binary
  • LittleEndian (if clickable, otherwise skip)

with NEXT to the Size interface, enter the dimensions which you find in the *vol.info file (less your_file.vol.info -> values NUM_X, NUM_Y, NUM_Z), the header is 0 for all HST and cutvol files.

On the next interface "Load As" you should click the preview button to check if your entered dimensions are ok. If you are not happy with the color range map (eg all the tomograph is white or black) try to play with the histogram values, a good estimate is always the intervall between 0 and 15.

On the next interface one can define a region of interest, please make sure that the memory required for loading your image is available (indicated at the bottom of the interface, green: ok, red: memory alert - out of memory).

Press again NEXT and FINISH! to start the loading process.

A complete development language for 3d images is also available called AVS. It is accessable on any Unix machine whithin the HMI intranet (Linux or Tru64) but the number of licenses available at one time is limited to 5 (license server).

For any volume file which you want to import a so called FLD file has to be created which is basically a headerfile in textformat which has to look like this

# AVS field file
# CREATOR: Alexander Rack
# DESCRIPTION: testz
# TYPE: 1
# SPACING: 5.000000e-06 5.000000e-06 5.000000e-06
# HISTORY: (null)
#
ndim=3   			 # number of dimensions in the field
dim1=687 			 # dimension of axis 1
dim2=717 			 # dimension of axis 2
dim3=50 			 # dimension of axis 3
nspace=3 			 # number of physical coordinates per point
veclen=1 			 # number of components at each point
data=byte 			 # data type (byte,integer,float,double)
field=uniform 			 # field type (uniform, rectilinear, irregular)
variable 1 file = 247.vol filetype=binary

Start it by entering start_express -nohw in any directory. Create a new application FILE -> New Application ("Multi-Window DataView" and "3D" should be enabled, rest disabled). Drag&Drop "ReadField" (DataImport folder) and "VolumeRender" (Visualization folder) in your application and connect the tools with the mouse, the result should look like this

In "MultiWindowApp" with the module "ReadField" you can import your volume file with browsing and selecting your FLD file. The result is displayed in the "Scene" window. Select the module "VolumeRender" to manipulate rendering etc. For more help and information see the detailed AVS online help.

f) how to use the BAM reconstruction software

One way to reconstruct tomographic images directly from DB files is to use the BAM software preko, to run it on nitomo1 create a cshell

csh

set the correct library path

setenv LD_LIBRARY_PATH

setenv LD_LIBRARY_PATH /hmi/bin/BAM/BESSY_SOURCES

and than

/hmi/bin/BAM/BESSY_SOURCES/preko_lf95 /absolute_path_to_db_file/your_file.db /absolute_path_to_bb_file/your_file.bb starting_slice ending_slice min_absorption_coefficient max_absorption_coefficient 0.0 x_axis_correction

Notes:

  • "starting_slice" and "ending_slice" are integers and have to be entered without decimal character (eg 0 20 etc)
  • rest of the parameters are floats and have to be entered with decimal character (eg 0.0 13.0 etc)
  • "x_axis_correction" is measured in pixels relative to the value set during the measurement
  • the resulting BB file contains the tomographic images as char 1byte and can easily be displayed with xdisp, it can also be imported in ImageJ, Visual Graphics Studio and AVS, please keep in mind here: values are stored as char 1byte, the BB files contain the BAM header with dynamical size, the size can be calculated by entering /hmi/bin/a4i/hmi2a4iL your_file.bb (see value "skip=..." [in bytes], leave the program with CTRL-C)

4. Histograms and char-volumes

The volume files done by PyHST or hst_slave are binary files, each voxel value is saved as FLOAT 4byte. 3d image analysis programs as well as rendering tools work with grey-scaled char 1byte data sets (which also helps lowering RAM and hard disc space consumption).

The first step for converting a FLOAT volume to a CHAR volume is creating a so called histogram, a plot of the voxel count vs. attenuation coefficients. Enter the directory with your volume file and then on nitomo1

/hmi/bin/ESRF/volStat your_volume_file > your_file_histogram

on any other Linux machine (nihaibel etc)

/chmi/nitomo1/bin/ESRF/volStat your_volume_file > your_file_histogram

or on any Tru64 machine (darling, nacha etc)

/chmi/nitomo1/bin/ESRF/volStat_alpha your_volume_file > your_file_histogram

The histograms can be plotted with "Origin" (Windows) or "xmgrace" (Unix) - both programs have an easy-to-use graphical interface. A bit quicker is to use gnuplot (Unix), start it by entering

gnuplot

and then

se lo y

pl "your_file_histogram"

You can save the file to postscript with

se te po

se ou "your_file.ps"

pl "your_file_histogram"

Leave the program with exit. A typical results looks like this

As you can see clearly the range where the voxel count function lives is the intervall [-10,70], so it is ok to project only these FLOAT 4byte values on the CHAR 1byte intervall [0, 255] (whole-numbered!!!). The idea is to identify the CHAR 1byte "0" value with the "-10" FLOAT 4byte. The "255" is then equal to "70" and all the whole-numbered values in between 0 and 255 are equal to ((70 - (-10) ) / 256)*whole_numbered_value . This of course leads to a reduction/loss of information.

This projection on CHAR 1byte is done with 2uchar, on nitomo1

/hmi/bin/ESRF/2uchar input.vol input.Nmin_abs_coeffXmax_abs_coeff.vol 0 min_abs_koeff max_abs_koeff

on any other Linux machine (nihaibel ...)

/chmi/nitomo1/bin/ESRF/2uchar input.vol input.Nmin_abs_coeffXmax_abs_coeff.vol 0 min_abs_koeff max_abs_koeff

on any other Tru64 machine (darling, nacha ...)

/chmi/nitomo1/bin/ESRF/2uchar_alpha input.vol input.Nmin_abs_coeffXmax_abs_coeff.vol 0 min_abs_koeff max_abs_koeff

Note:

One check you should do is if the noise is oversampled. Load one slice with ImageJ and select an area with no absorption (pore, vacuum etc - use area tools rectangle, circle etc). Select ANALYZE -> SET MEASUREMENTS and enable "Standard Deviation" -> OK. Now ANALYZE -> MEASURE, the "StdDev" value shoud be bigger than (max_abs_coeff - min_abs_coeff)/256 !

The CHAR 1byte volume files can be rendered with Visual Graphics Studio as well as AVS - just keep in mind to use 1byte CHAR instead of 4byte FLOAT as importing parameter.

when you are finished with the converting to char do not forget to remove your FLOAT volume files to lower hard disc consumption

5. Backup

For backups currently DVD burners (multi-norm, double-layer), a Tandberg DLT8000 tapedrive and an Ultrium/HP LTO2 tapedrive are available. DVD burners are located in rooms LR241 (nitomo2 - single-layer, nitomo1 - double-layer), LR235, LR239 and LR206. On Windows systems files can be transfered to the corresponding PCs with "sftp" and burned with "NeroBurningRom". On Linux machines (nitomo1, nitomo2 ...) use scp for copying and /opt/kde3/bin/k3b or /usr/X11R6/bin/xcdroast for burning.

The DLT8000 is connected with the SCSI card at nitomo2 (located in LR241) - locked! The key is available from Bajorat, Haibel or Rack. Each tape can store 40.000.000.000 bytes (37.25 GB) in non-compressed mode, compressed it can store up to 80.000.000.000 bytes depending on your data. To load or unload a tape follow these instructions

The LTO2 is connected with the SCSI card at nitomo3 (located in LR241) - locked! The key is available from Bajorat, Haibel or Rack. Each tape can store 200.000.000.000 bytes (187 GB) in non-compressed mode, compressed it can store up to 400.000.000.000 bytes depending on your data. To load a tape slide it gently in the drive until the automatic loading process takes over. To unload the tape just push the eject-button.

To restore data from a tape: load the tape, enter the directory of your choice and run (for more security it is recommended you enable write protection on your tape)

tar xvf /dev/st0

(or tar xvf /dev/nst0 - no rewind performed after tar reaches EOF)

If you get an error message like

tar: /dev/nst0: Kann read nicht ausführen.: Nicht genügend Hauptspeicher verfügbar
tar: Am Anfang des Mediums, beende jetzt.
tar: Nicht behebbarer Fehler: Programmabbruch.
then please use

tar -x -v --blocking-factor=BLOCK_SIZE -f /dev/nst0

the BLOCK_SIZE value should be noted on your DLT tape (tar -b 5120 ...) if not than try 128.

To display the current status of the tapedrive (eg the blocksize currently used) enter

mt -f /dev/st0 status

to perform rewind

mt -f /dev/st0 rewind

to enable/disable compression (only root can do that via commands, if you are not root than use the buttons directly at the tape machine)

mt -f /dev/st0 datcompression 1/0

change the block size used (entering 0 means dynamic block size)

mt -f /dev/st0 setblk [int]
(if only root can do that via commands - ask Bajorat, Haibel or Rack)

more options are listed with man mt .

For optimal performance it is recommended to use a blocksize of 64k (mt -f /dev/st0 setblk 65536) and 5120 bytes with tar (tar -b 5120 ...). Please note the blocksizes used on your tape otherwise it is not possible to restore the data with tar!

To backup data to a free tape (available from Bajorat, Haibel or Rack): load the tape, enter the directory of your choice and use tar with the following syntax

tar cfv /dev/nst0 your_source |& tee -a /hmi/logfiles/tape_indexcount_your_description

or

tar -c -v -h -b 5120 -f /dev/nst0 your_source |& tee -a /hmi/logfiles/tape_indexcount_your_description where "-b 5120" sets the tar-blocksize to use and "-h" makes tar following symbolic links.

Please create a logfile for each tape you do by using the syntax above. Please use continuous tape index.

If tar crashes with a message like "Unrecoverable error: 0 out of 202334 bytes haven been written ..." ("tar: /dev/nst0: Nur 0 von 10240 Bytes geschrieben. tar: Nicht behebbarer Fehler: Programmabbruch. ") please contact Rack, Bajorat or Haibel or check if the size of the data you want to store is larger than (in case of DLT8000) 40.000.000.000 (note: 40*10^9 bytes and NOT 40 GB)! Recommended size is smaller 37 GB!

If you are not able to restore a tape it can be due to conflicting blocksizes (eg the tape is written with 64k while your system is set to use dynamic blocksizes, usally only those two options are chosen) - switch (as root) the blocksize with

mt -f /dev/st0 setblk 0
(dynamic blocksize)

or

mt -f /dev/st0 setblk 65536
(64k blocksize)

Last updated: 3rd august 2005, Alexander Rack