pynx-holotomo-id16b

Script for single or multi-distance holo-tomography reconstruction from data recorded on the ESRF ID16B beamline.

usage: pynx-holotomo-id16b [-h] --data DATA --delta_beta DELTA_BETA [--nz NZ] [--planes PLANES [PLANES ...]] [--binning BINNING] [--first FIRST] [--last LAST] [--step STEP] [--remove_spikes REMOVE_SPIKES] [--double-flat-field [DOUBLE_FLAT_FIELD ...]] [--distorted [DISTORTED]] [--normalise] [--nrj NRJ]
                           [--distance DISTANCE [DISTANCE ...]] [--pixel_size PIXEL_SIZE [PIXEL_SIZE ...]] [--sx0 SX0] [--pixel_size_detector PIXEL_SIZE_DETECTOR] [--cx CX] [--sx SX [SX ...]] [--halftomo {left,right}] [--rhapp RHAPP]
                           [--align {fft,fft_diff,rhapp,fft_spline3,fft_spline4,fft_spline5,fft_spline6,fft_diff_spline3,fft_diff_spline4,fft_diff_spline5,fft_diff_spline6,fft_poly2,fft_poly3,fft_poly4,fft_poly5,fft_poly6,fft_diff_poly2,fft_diff_poly3,fft_diff_poly4,fft_diff_poly5,fft_diff_poly6,fft_median3,fft_median5,fft_median7,fft_diff_median3,fft_diff_median5,fft_diff_median7}]
                           [--align_interval ALIGN_INTERVAL] [--align_max_shift ALIGN_MAX_SHIFT] [--align_fourier_low_cutoff ALIGN_FOURIER_LOW_CUTOFF] [--align_fourier_high_cutoff ALIGN_FOURIER_HIGH_CUTOFF] [--reference_plane {1,2,3,4}] [--motion] [--algorithm ALGORITHM] [--padding PADDING]
                           [--pad_method {edge,mean,median,reflect,reflect_linear,reflect_erf,reflect_sine,symmetric,wrap}] [--stack_size STACK_SIZE] [--obj_smooth OBJ_SMOOTH] [--obj_inertia OBJ_INERTIA] [--obj_min OBJ_MIN] [--obj_max OBJ_MAX] [--beta BETA] [--nb_probe {1,2,3,4}]
                           [--tomo_rot_center TOMO_ROT_CENTER] [--tomo_rot_center_near TOMO_ROT_CENTER_NEAR] [--tomo_cor_method {radios_sliding_window,radios_growing_window,global,phase_registration,phase_registration_diff,phase_registration_high,phase_registration_diff_high}]
                           [--sino_filter {ram-lak,shepp-logan,cosine,hamming,hann,tukey,lanczos,none}] [--prefix_output PREFIX_OUTPUT] [--save_phase_chunks] [--save_edf] [--save_fbp_vol {hdf5,tiff,cuts}] [--nxtomo] [--save_3sino] [--ngpu NGPU] [--slurm]
                           [--slurm_partition [{gpu,p9gpu,gpu-long,p9gpu-long,low-gpu,low-p9gpu} ...]] [--slurm_nodelist SLURM_NODELIST] [--slurm_time SLURM_TIME] [--max_mem_chunk MAX_MEM_CHUNK] [--profile]

Named Arguments

--data

Path to the dataset directory, e.g. path/to/sample_1, For nz>1 this should be one of the distances directories. This can also use a more explicit format with e.g. path/to/sample_%02d_ in case the usual naming changed.

--delta_beta

delta/beta value for CTF/Paganin

Input parameters

--nz

Number of distances (planes) for the analysis

--planes

Planes to use for the analysis. This is normally set by –nz, e.g. ‘–nz 4’ is equivalent to ‘–planes 1 2 3 4’. This option can be used to exclude specific planes e.g. the first distance using ‘–planes 2 3 4’

--binning

Binning parameter.

Default: 1

--first

Index of the first projection to analyse. Default=0 (first available projection)

Default: 0

--last

Index of the last (included) projection to analyse. Defaults to the last recorded projection, automatically detected.

--step

Step to pick the projections to analyse. Defaults to the binning value. Can also be >1 to distribute projections to independent jobs for phasing, or to separate odd and even frames. For example using ‘–step 2 –first 0’ and ‘–step 2 –first 1’ will respectively use even and odd frames.

Pre-processing

--remove_spikes

Replace any pixel which differs from the median-filtered (3x3) image by more than this (relative) value. Typical value 0.04. Filtering is done after dark subtraction.

Default: 0

--double-flat-field, --double_flat_field

Apply a double flat-field correction, i.e. normalise the radios by their average value after normalisation to the white/empty beam images. Optional filtering options can be given, e.g. ‘–double-flat-field 0.1 0.05’: the first number corresponds to the high-pass filter which will filter out frequencies below 0.1 in the 2D imageplane, and the second is a low-pass filter along the projections axis (this second filter requires much longer 3D calculations ).By default the values are (0, 0), which means that the radiosare normalised by <radios/ref>_z (where the average is taken along the projections index). If the high-pass filter is used, then the normalisation is instead 1+F(radios/ref) where F is the filter used. Good values can be between 0.001 to 0.05 for the first (relative) frequency (larger values removemost of the correction), and up to 0.1 for the second (0.1 would remove ring artefacts that span about 1/10th of a full ring.)

--distorted

Correct for the distortion of frames vs reference images. Can be used for local tomography. Optionally, one argument with the tile size can be given (default is 100 if ‘–distorted’ is passed without argument). Valid values range from 40 to 400.

Default: 0

--normalise

Use this keyword to normalise the observed projections by the reference images, immediately upon loading. The normalised radios are scaled to keep the average intensity, so that the statistical figure of merits are unaffected. Note that currently (EXPERIMENTAL), this triggers the pre-shifting of all frames before phasing, and so disables probe optimisation.

Default: False

Experimental parameters

--nrj

Energy (keV). By default will be read from the info or edf files.

--distance

Sample-detector distance(s) (m), after magnification. Multiple values should be given if nz>1 using ‘–distance 1e-3 1.2e-3 1.5e-3 1.9e-3’ . By default will be automatically calculated from .info and/or motor positions.

--pixel_size

Pixel size(s) (m), for each distance, after magnification. Multiple values should be given if nz>1 using ‘–pixel_size 1e-8 1.2e-8 1.6e-8 2.2e-8’ . By default this is computed from .info and/or edf files. Will be multiplied by binning.

--sx0

focus offset parameter in mm: this is used to compute the magnified pixel sizes and sample-detector distances will be calculated from cx and sx. By default this value is automatically deduced from the data files, using sx, cx, and the magnified and non-magnified pixel sizes.

--pixel_size_detector

Detectr pixel size (m), before magnification. This can be used only in combination with sx0, to computethe magnified distances. If not given, this is read from the ‘optic_used’ field in the .info or edf files.

--cx

detector motor position (mm), used if sx0 is also given. By default the value is read from edf files, so this should only be used if there was something wrong in the recorded positions.

--sx

sample motor position (mm), used only if sx0 is also given. Multiple values should be given (–sx 3 4 5 6) if nz>1. By default the value is read from edf files, so this should only be used if there was something wrong in the recorded positions.

--halftomo

Possible choices: left, right

Use this option for a half-tomo dataset. This will trigger the search for the center of rotation on the correct side, and expand the field of view during the tomography reconstruction with Nabu.

Alignment parameters

--rhapp

Filename for the rhapp alignment matrix written by holoCT. If set, this automatically sets the alignment method to using the rhapp matrix, unless –align is also used (the rhapp matrix can then still be used for comparison).

--align

Possible choices: fft, fft_diff, rhapp, fft_spline3, fft_spline4, fft_spline5, fft_spline6, fft_diff_spline3, fft_diff_spline4, fft_diff_spline5, fft_diff_spline6, fft_poly2, fft_poly3, fft_poly4, fft_poly5, fft_poly6, fft_diff_poly2, fft_diff_poly3, fft_diff_poly4, fft_diff_poly5, fft_diff_poly6, fft_median3, fft_median5, fft_median7, fft_diff_median3, fft_diff_median5, fft_diff_median7

Alignment method, ‘fft’ (Fourier cross-correlation), ‘fft_splineN’ (same as fft plus a 3rd-order spline fit with N=3-6 knots), ‘fft_polyN’ (same as fft plus a Nth order polynomial fit (N=2-6), ‘fft_medianN’ (same as fft with only a median filter of width N=3,5 or 7)), ‘fft_diff’, ‘fft_diff_splineN’, ‘fft_diff_polyN’, ‘fft_diff_medianN’: same as the previous methods, but align using the difference between theprojection and the rolling 3-projection average-this approach is very robust but incompatible with random motion. The default is ‘fft_diff_spline3’ or ‘fft_spline3’ if –motion is used. Alternatively, ‘rhapp’ will be used automatically if –rhapp is given.

--align_interval

if > 1, align only every N projection, and interpolate for the other projections. N>1 can be useful if loading & alignment is slower than algorithms when using multiple GPUs.

Default: 1

--align_max_shift

Maximum shift (+/-) expected in pixels between projections. This is usedto limit the shift range and to zero-pad when aligning. The value will be divided by the binning value, if any.

Default: 200

--align_fourier_low_cutoff

Cutoff (default: None), a value (0< <=0.05) to cutoff low frequencies during Fourier registration.

--align_fourier_high_cutoff

Cutoff (default: None), a value (0.25<= < 0.5) to cutoff high frequencies during Fourier registration. Can be useful for noisy datasets.

--reference_plane

Possible choices: 1, 2, 3, 4

Reference plane for the alignment, for nz>1. Default is the first plane. Numbering begins at 1, same as for the –planes argument

--motion

Take into account the random motion ‘correct.txt’ data during alignment

Default: False

Reconstruction parameters

--algorithm

Algorithm string, executed from right to left. It is recommended to use the following aliases for standard algorithm chains:

• paganin-norm: equivalent to AP**5,delta_beta=0,AP**10,Paganin (when using –normalise, which makes long iterations ineffective)

• ctf-norm: same as paganin-norm, but with CTF

• paganin-short: eq. to AP**5,obj_smooth=0,delta_beta=0,AP**10,Paganin,obj_smooth=4

• ctf-short: same as paganin-short, but with CTF

• paganin-long: eq. to AP**20,obj_smooth=0,delta_beta=0,AP**40,Paganin,obj_smooth=4

• ctf-long: same as paganin-long, but with CTF

Note that in the above, the ‘obj_smooth’ is a regularisation term which forces the probe to incorporate as much as possible of the high frequencies, thus reducing ring artefacts in the object. It does not smooth the final object as long as 5-10 cycles without smoothing are done at the end.

Examples of explicit algorithms:

• AP**5,delta_beta=0,AP**10,Paganin

• AP**10,obj_smooth=0.5,AP**20,obj_smooth=8,CTF

• AP**10,obj_smooth=0.5,Paganin

• AP**10,obj_smooth=0.5,DRAP**10,obj_smooth=1,CTF

Finally, note that large number of cycles (e.g. >70 AP for nz=4 and ~3200 projections and padding=200) will take more than one hour,and will require using either the ‘gpu-long’ or ‘low-gpu’ partition,with the –slurm_partition option.

Default: “AP**5,delta_beta=0,AP**10,CTF”

--padding

Number of pixels to pad on every side of the array. 20% of the size is normally enough. The value will be rounded to the closest value which allows a radix FFT. 2 can be given as a special value and will correspond to a doubling of the array size, i.e. a padding of ny/2 and nx/2 along each dimension (not recommended, this wastes memory and computing time). NOTE: this is only the padding used during phasing - when computing the filtered back-projection, the padding used always doubles the array size.

Default: 0

--pad_method

Possible choices: edge, mean, median, reflect, reflect_linear, reflect_erf, reflect_sine, symmetric, wrap

Padding method. See methods description in numpy.pad() online doc, with ‘reflec’ additional methods, using a linear combination of wrapped reflected arrays across the border, to guarantee a continuity acrossthe edges of the padded array.

Default: “reflect_linear”

--stack_size

Stack size-how many projections are stored simultaneously in memory. This is normally automatically adapted to the available GPU memory.

--obj_smooth

Object smoothing parameter in pixels. Can be overridden in the algorithm string. Requires obj_inertia>0 to work. Note that it does not smooth directly the object by a gaussian of the given sigma, but rather regularises the updated object to be close to the previous gaussian-blurred iteration.

Default: 0

--obj_inertia

Object inertia parameter. Can be overridden in the algorithm string. Required > 0 to exploit smoothing, typical values 0.1 to 0.5.

Default: 0.1

--obj_min

Object minimum amplitude (e.g. 1)

--obj_max

Object maximum amplitude (e.g. 1)

--beta

Beta coefficient for the RAAR or DRAP algorithm. Not to be mistakenwith the refraction index beta

Default: 0.9

--nb_probe

Possible choices: 1, 2, 3, 4

Number of coherent probe modes. If >1, the (constant) probe mode coefficientswill linearly vary as a function of the projection index. EXPERIMENTAL

Default: 1

Tomography reconstruction parameters

--tomo_rot_center

Rotation centre (pixels), not taking into account binning. By default, this is determined automatically.

--tomo_rot_center_near

Estimated position for the rotation centre (pixels), not taking into account binning. This value is passed to Nabu’s CoR methods for CoR estimation (near_pos).

--tomo_cor_method

Possible choices: radios_sliding_window, radios_growing_window, global, phase_registration, phase_registration_diff, phase_registration_high, phase_registration_diff_high

Method used for the determination of the centre of rotation, either sliding/growing window on radios (from Nabu), ‘sino’ using Nabu’s SinoCOR, or ‘phase_registration’ using cross-correlation of phased projections at +180°

Default: “radios_sliding_window”

--sino_filter

Possible choices: ram-lak, shepp-logan, cosine, hamming, hann, tukey, lanczos, none

Name for the sinogram 1D filter before back-projection. The filtering will be done using the padded array if padding is used.

Default: “ram-lak”

Output parameters

--prefix_output

Prefix for the output (can include a full directory path), Defaults to data_prefix_resultNN

--save_phase_chunks

Save the phased projections to an hdf5 file. This is deprecated, use –nxtomo instead

Default: False

--save_edf

Save the phased projections individual edf files.

Default: False

--save_fbp_vol

Possible choices: hdf5, tiff, cuts

Save the reconstructed volume, either using the hdf5 format (float16), uint16 tiff (single file if volume<4GB, individual slices otherwise),or only with a png with the 3 cuts (mostly for testing)

Default: False

--nxtomo

Save the phased projections using the NXTomo format.

Default: False

--save_3sino

Save the 3 sinograms at 25, 50 and 75% of height for testing.

Default: False

Job parameters

--ngpu

number of GPUs the job should be distributed to (default=2for a slurm job, otherwise defaults to 1)

--slurm

If given, the job will be submitted as an ESRF slurm job (p9gpu)

Default: False

--slurm_partition

Possible choices: gpu, p9gpu, gpu-long, p9gpu-long, low-gpu, low-p9gpu

Specify the slurm partition(s) to use (default is gpu,p9gpu, 1 hour max)

Default: [‘gpu’, ‘p9gpu’]

--slurm_nodelist

Specify a nodelist for the slurm job submission (mostly for debugging or very specific needs e.g. memory). The string will be passed directly to sbatch –nodelist=…

--slurm_time

Specify a time limit for the slurm job (mostly for debugging). The string will be passed directly to sbatch –time=… . By default the time limit is automatically chosen.

--max_mem_chunk

Maximum memory (GB) available per chunk (ngpu chunks are processed in //).The actual memory used should be about 1.5x this value. This should be keptreasonably low because the process uses a lot of shared memory, and theanalysis can slow down significantly if too much is used.

Default: 64

--profile

Enable GPU profiling (for development only)

Default: False

Example usage:

• quick (binned x2) 4-distance reconstruction, Paganin, save tiff volume:

  pynx-holotomo-id16b --data data/alcu_25nm_8000adu_1_ --delta_beta 530 --save_fbp_vol tiff --algorithm AP**5,delta_beta=0,AP**10,Paganin --nz 4 --padding 100 --binning 2 --slurm

• 4-distance reconstruction, CTF, save tiff volume, distortion correction:

  pynx-holotomo-id16b --data data/alcu_25nm_8000adu_1_ --delta_beta 530 --save_fbp_vol tiff --algorithm AP**5,delta_beta=0,AP**20,CTF --nz 4 --padding 200 --distorted --slurm

• 4-distance reconstruction, CTF, save tiff volume, distortion correction, use normalised projections (disables probe optimisation):

  pynx-holotomo-id16b --data data/alcu_25nm_8000adu_1_ --delta_beta 530 --save_fbp_vol tiff --algorithm AP**5,delta_beta=0,AP**20,CTF --nz 4 --padding 200 --distorted --normalise --slurm

• single distance reconstruction, Paganin, save tiff volume:

  pynx-holotomo-id16b --data data/alcu_25nm_8000adu_1_ --delta_beta 530 --save_fbp_vol tiff --algorithm AP**5,delta_beta=0,AP**20,Paganin --nz 1 --padding 200 --slurm

• long 4-distance optimisation using progressive smoothing (improves artefact removal, without actually smoothing the object), distortion correction:

  pynx-holotomo-id16b --data data/alcu_25nm_8000adu_1_ --delta_beta 530 --save_fbp_vol tiff --algorithm AP**5,delta_beta=0,obj_smooth=0,AP**20,obj_smooth=1,AP**20,obj_smooth=4,AP**20,obj_smooth=8,Paganin --nz 4 --padding 200 --distorted --slurm