Ctrl+K

Table of Contents

pynx-ptycho-id16a-nf#

Script to perform a ptychography analysis on NEAR FIELD data from ID16A@ESRF

usage: pynx-ptycho-id16a-nf [-h] [--scan SCAN [SCAN ...]] [--max_frame MAXFRAME] [--max_size MAXSIZE] [--modulo_frame MODULOFRAME [MODULOFRAME ...]] [--mask LOADMASK] [--mask_iobs_max MASK_IOBS_MAX] [--roi ROI [ROI ...]] [--rebin REBIN] [--xy XY]
                            [--detector_orientation DETECTOR_ORIENTATION DETECTOR_ORIENTATION DETECTOR_ORIENTATION] [--flatfield FLATFIELD] [--dark DARK] [--dark_subtract [DARK_SUBTRACT]] [--orientation_round_robin] [--nrj NRJ] [--pixel_size PIXELSIZE] [--near_field] [--detector_distance DETECTORDISTANCE]
                            [--algorithm ALGORITHM] [--nb_run NBRUN] [--run0 RUN0] [--no_rerun] [--no_auto_center] [--auto_center {0,1}] [--center_probe_n CENTER_PROBE_N] [--center_probe_max_shift CENTER_PROBE_MAX_SHIFT] [--dm_loop_obj_probe DM_LOOP_OBJ_PROBE] [--use_direct_beam]
                            [--ml_obj_regularisation ML_OBJ_REGULARISATION] [--obj_smooth OBJ_SMOOTH] [--obj_inertia OBJ_INERTIA] [--probe_smooth PROBE_SMOOTH] [--probe_inertia PROBE_INERTIA] [--pos_mult POS_MULT] [--pos_max_shift POS_MAX_SHIFT] [--pos_min_shift POS_MIN_SHIFT] [--pos_threshold POS_THRESHOLD]
                            [--background_smooth BACKGROUND_SMOOTH] [--dm_alpha DM_ALPHA] [--raar_beta RAAR_BETA] [--live_plot] [--verbose VERBOSE] [--fig_num FIG_NUM] [--save_prefix SAVEPREFIX] [--output_format {cxi,npz,nxtomo}] [--cxi_output {object_probe,object_phase,object,probe}]
                            [--remove_obj_phase_ramp] [--save {final,all}] [--data2cxi [{crop}]] [--movie] [--obj_max_pix OBJ_MAX_PIX] [--obj_margin OBJ_MARGIN] [--load LOAD] [--load_probe LOADPROBE] [--load_pixel_size LOADPIXELSIZE] [--probe PROBE] [--defocus DEFOCUS] [--rotate ROTATE] [--object OBJECT]
                            [--multiscan_reuse_ptycho [MULTISCAN_REUSE_PTYCHO]] [--mpi {split,scan,multiscan}] [--mpi_split_nb_overlap MPI_SPLIT_NB_OVERLAP] [--mpi_split_nb_neighbour MPI_SPLIT_NB_NEIGHBOUR] [--gpu GPU] [--stack_size STACK_SIZE] [--instrument INSTRUMENT] [--share_probe [{-1,0,1,2,3,4,5,6}]]
                            --data DATA --meta META [--data_ref DATA_REF] [--ptycho_motors PTYCHOMOTORS [PTYCHOMOTORS ...]] [--delta_beta DELTA_BETA] [--adu_scale ADU_SCALE] [--tomo_motor TOMO_MOTOR] [--padding PADDING] [--save_plot [{object_phase,object_rgba}]]

Input parameters#

--scan

scan number e.g. –scan 5 to extract a given scan number from a series. Alternatively a list or range of scans can be given using “–scan 12 23 45”or –scan “range(12,25)” (note the quotes, necessary when using range with ()).

--max_frame, --maxframe

maximum number of frames to import

--max_size, --maxsize

Maximum frame size to use, while keeping center of gravity of diffraction in the cut frame center.

Default: 10000

--modulo_frame, --moduloframe

Instead of taking all sequential frames, load one every N. If two numbers are given (–moduloframe N1 N2), N2 must be smaller than N1 and the frames loaded will be those with i % N1 == N2.

--mask, --loadmask, --load_mask

Specify th mask to use for detector data, which should have the same 2Dshape as the raw detector data.This should be a boolean or integer array with good pixels=0 and bad ones>0(values are expected to follow the CXI convention)Acceptable formats/values: - “mask.npy”, “mask.npz” (the first data array will be used) - “mask.edf” or “mask.edf.gz” (a single 2D array is expected) - “mask.h5:/entry_1/path/to/mask”: hdf5 format with the full path to the 2D array. hdf5 is also accepted as extension. - “maxipix”: if this special name is entered, the masked pixels will be rows and columns multiples of 258+/-3

--mask_iobs_max, --maskiobsmax

if given, all pixels with an observed intensity >= the given value are masked. This is applied frame-by-frame.

--roi

Region-of-interest to be used for actual inversion. This requires 4 valuesfor xmin xmax ymin ymax. The area is taken with python conventions, i.e. pixels with indices xmin<= x < xmax and ymin<= y < ymax.Additionally, the shape of the area must be square, and n=xmax-xmin=ymax-ymin must also be a suitable integer numberfor OpenCL or CUDA FFT, i.e. it must be a multiple of 2 and the largest number inits prime factor decomposition must be less or equal to the largest valueacceptable by vkFFT for a radix transform(<=13).If n does not fulfill these constraints,it will be reduced using the largest possible integer smaller than n.This option supersedes “maxsize” unless roi=”auto”. Other possible values: - “auto”: automatically selects the roi from the center of mass and the maximum possible size. [default] - “all” or “full”: use the entire, uncentered frames. Only useful for pre-processed data. Cropping may still be performed to get a square and FFT-friendly size. Near-field should default to “full”

Default: “full”

--rebin, --binning, --bin

the experimental images can be rebinned (i.e. a group of n x n pixels is replaced by a single one whose intensity is equal to the sum of all the pixels). Binning is performed last: the ROI, mask, background, pixel size should all correspond to full (non-rebinned) frames.. This is recommended if the size of the probe (including tails) is smaller than half of the Fourier (the probe array) window (then use --rebin 2): reconstruction will be 4x faster without any loss of resolution. It is also recommended if the extent of the scanned area is smaller than the size of the Fourier (probe) array.

--xy

expression to be used for the XY positions (e.g. “-x,y”,…), to compute thevalues from the raw input (scaling, flipping, etc..). Mathematical operations can also be used, e.g.: --xy=0.5*x+0.732*y,0.732*x-0.5*y

--detector_orientation, --detectororientation

three optional flags which, if 1, will do in this order: array transpose (x/y exchange), flipud, fliplr. The changes also apply to the mask

--flatfield, --flat_field, --flat

flatfield path for the correction to be applied to the detector data. The array must have the same shape as the frames, which will be multiplied by this correction. Acceptable formats: flat.npy or flat.npz (the first data array will be used), flat.edf or flat.edf.gz (a single 2D array is expected), “flat.h5:/entry_1/path/to/flat” (hdf5 format with the full path to the 2D array, hdf5 is also accepted as extension), flat.mat (from a matlab file. The first array found is loaded)

--dark

the dark correction file (incoherent background). The array must have have the same shape as the frames, which will be multiplied by this correction. Acceptable formats: flat.npy or flat.npz (the first data array will be used), flat.edf or flat.edf.gz (a single 2D array is expected), “flat.h5:/entry_1/path/to/flat” (hdf5 format with the full path to the 2D array, hdf5 is also accepted as extension), flat.mat (from a matlab file. The first array found is loaded)

--dark_subtract

use this option to subtract the dark from the observedintensity. This is normally discouraged, as it will mess up the Poisson statisticsof the observed data, but it can be useful in the case of very high background.If given as a simple keyword, the dark is subtracted. If given with a float,the dark will be multiplied by this factor before subtraction

Default: False

--orientation_round_robin, --orientationroundrobin

use this option to test all possible combinations of xy and detector_orientation to find the correct detector configuration. There are 64 possibilities, with a redundancy of 8.

Default: False

--nrj, --energy

X-ray beam energy in keV (not needed if read from the data files)

--pixel_size, --pixelsize

detector pixel size in meters (not needed if read from the data files)

--near_field, --nearfield

Use this for near-field ptycho optimisation. It is normally automatically triggered by the script called, but may be useful in some cases (e.g. near-field data in CXI format)

Default: True

--detector_distance, --detectordistance, --distance

Detector distance in meters. Mandatory unless included in the data file.

Algorithms#

--algorithm

algorithm chain used for the optimization. This chain:

  • is divided in “steps” separated by commas, e.g: ML**50,DM**100

  • is interpreted from right to left, as a mathematical operator to an object on the right-hand side

  • should not contain any space, unless the whole string is given between quotes (“”)

There are two types of commands in the chain:

  1. commands which change basic parameters or perform some analysis:

  • probe=1 or 0: activate or deactivate the probe optimisation (by default only the object is optimised)

  • object=1 or 0: activate or deactivate the object optimisation

  • background=1 or 0: activate or deactivate the background optimisation. When set to 1, this will initialise the background to at least 1e-2 to enable the background optimisation.

  • background_smooth=3: gaussian sigma for smoothing the updated background [default:3, large values are possible and will use FFT convolution]

  • position=N or 0: activate or deactivate (0) position optimisation every N cycles (preferably for AP or ML, some prior convergence is needed. Can also work with DM but is not recommended). Recommended value is every 5 cycles

  • pos_mult=1: multiplier for the calculated position shift. Can be used to accelerate convergence or make it more cautious. (default:5, suitable for an object with reasonable contrast)

  • pos_max_shift: maximum shift of position update in pixels between iterations (default:2)

  • pos_min_shift: minimum shift of position update in pixels between iterations (default:0)

  • pos_threshold: if the integrated norm of the object gradient multiplied by the probe lower than the average value (for all positions) multiplied by this threshold, the position is not changed. This allows to avoid updating positions in areas where the object is flat, and sensitivity to shifts is low. [default:0.2]

  • nbprobe=3: change the number of modes for the probe (can go up or down)

  • regularization=1e-4: setting the regularization parameter for the object, to penalize local variations in ML runs and smooth the solution

  • obj_smooth=1.5 or probe_smooth=1.0: these parameters will partially smooth the object and probe, softening the resulting arrays. This applies to DM and AP algorithms.

  • obj_inertia=0.01 or obj_probe=0.001: these parameters set the inertia of the object and/or probe update, yielding more stable result. This applies to DM and AP algorithms.

  • ortho: will perform orthogonalisation of the probe modes. The modes are sorted by decreasing intensity.

  • analysis: perform an analysis of the probe (propagation, modes). Useful combined with “saveplot” to save the analysis plots

  1. operators which indicate the actual algorithm to apply to the object and probe:

  • AP: alternate projections. Slow but converging algorithm

  • DM: difference map. Fast early convergence, oscillating after.

  • ML: maximum likelihood conjugate gradient (Poisson-noise). Robust, converging, for final optimization.

These operators can be combined mathematically, e.g.:

  • DM**100: corresponds to 100 cycles of difference map

  • ML**40*DM**100: 100 cycles of DM followed by 40 cycles of ML (note the order)

Example algorithms chains:

  • --algorithm ML**40,DM**100,probe=1: activate probe optimisation, then 100 DM and 40 ML

  • --algorithm ML**100,DM**200,nbprobe=3,ML**40,DM**100,probe=1,DM**100: first DM with object update only, then 100 DM also updating the probe, then use 3 probe modes and do 100 DM followed by 40 ML

  • --algorithm ML**100*AP**200*DM**200,probe=1: 200 DM then 200 AP then 100 ML (one step)

  • --algorithm "(ML**10*DM**20)**5,probe=1": repeat 5 times [20 cycles of DM followed by 5 cycles of ML] (note the quotes necessary for the parenthesis)

Default: “ML**50,DM**100,probe=1”

--nb_run, --nbrun

number of times to run the optimisation

Default: 1

--run0

number for the first run (can be used to overwrite previous run results)

--no_rerun, --norerun, --no_re_run

if this option is used, the scan will be skipped if the result file already exists. This allows to run batch jobs for multiple scans with low priority (and which can be preempted) and restart them without reprocessing scans. [default: save to a new run if saveprefix has a %d field for the run number, and overwrite otherwise]

Default: False

--no_auto_center, --noautocenter

Use this option to disable auto-centering of object and probe oftereach optimisation step. This is done by default to avoid drifts.

Default: False

--auto_center, --autocenter

Possible choices: 0, 1

Use this option to disable (0) auto-centering of object and probe after each optimisation step. This is done by default to avoid drifts (notably during DM).

Default: 1

--center_probe_n

During DM, check for a probe drift every N cycles given by this argument.Also see center_probe_max_shift. Ignored for near field ptycho.

Default: 5

--center_probe_max_shift

During DM, when checking for a probe drift, the object and probe positions will be corrected if the center deviates by more the given number of pixels.Ignored for near field ptycho.

Default: 5

--dm_loop_obj_probe

during DM, when both object and probe are updated, it may be better to loop the object and probe update for a more stable optimisation (but slower).

Default: 1

--use_direct_beam, --usedirectbeam

if used, and a reference frame with the direct beam is given, itwill also be used for the optimisation, giving an absolute reference for the probe

Default: False

--ml_obj_regularisation, --regularization

object regularisation factor to smooth during ML

Default: 0

--obj_smooth, --objsmooth

object smoothing width during projection algorithms. Must be associated with obj_inertia>0 for effect.

Default: 0

--obj_inertia, --objinertia

object inertia (0-1) during projection algorithms.

Default: 0.05

--probe_smooth, --probesmooth

probe smoothing width during projection algorithms. Must be associated with probe_inertia>0 for effect.

Default: 0

--probe_inertia, --probeinertia

probe inertia (0-1) during projection algorithms.

Default: 0.005

--pos_mult, --posmult

multiplier for the calculated position shift. Can be used to accelerate convergence or make it more cautious. (default:5, suitable for an object with reasonable contrast)

Default: 5

--pos_max_shift, --posmaxshift

maximum shift of position update in pixels between iterations

Default: 1

--pos_min_shift, --posminshift

minimum shift of position update in pixels between iterations - this can be used to ignore small shifts (due to noise)

Default: 0

--pos_threshold, --posthreshold

if the integrated norm of the object gradient multiplied by the probe is lower than the average value (for all positions) multiplied by this threshold, the position is not changed. This allows to avoid updating positions in area where the object is flat, and sensitivity to shifts is low.

Default: 0.2

--background_smooth, --backgroundsmooth

gaussian sigma to smooth the updated background

Default: 3

--dm_alpha, --dmalpha

alpha parameter to mix a small amount of AP with DM

Default: 0.02

--raar_beta, --raarbeta

RAAR beta parameter

Default: 0.9

Display#

--live_plot, --liveplot

liveplot during optimisation

Default: False

--verbose

print evolution of llk (and display plot if ‘liveplot’ is set) every N cycle

Default: 50

--fig_num

figure number to display object and probe

Default: 100

Output#

--save_prefix, --saveprefix
prefix to save the optimized object and probe (as a .cxi or .npz file) and optionally image (png). Use “saveprefix=none” to disable saving (e.g. for tests). Several format options can be used, including the “run” number, the “scan” number, the data directory “data_dir” or thedata prefix “data_prefix” (data filename without extension). Examples:

–save_prefix ResultsScan{scan:04d}/Run{run:04d} : the default, using the scan and run number to identify the result –save_prefix ResultsScan%04d/Run%04d : same as the previous format, but using old-style formatting (deprecated) –save_prefix ResultsScan{scan:04d} : only use the scan number in the output format (new runs will overwrite the previous ones –save_prefix {data_prefix}_run{run:04d} : combining the original data name and the run number –save_prefix {data_dir}/{data_prefix}_results/Run{run:04d} : saving along the original data file.

Default: “ResultsScan{scan:04d}/Run{run:04d}”

--output_format, --outputformat

Possible choices: cxi, npz, nxtomo

choose the output format for the final object and support.Note: nxtomo is only for ptycho-tomo, and requires the rotation angle of each projection. This is only supported by the ID16A NFP runner.

Default: “cxi”

--cxi_output, --cxioutput

Possible choices: object_probe, object_phase, object, probe

Choice of data included in the cxi result file.This only affects the large object and probe arrays. Ignored for –output_format nxtomo. Choices: - object_probe: all object and probe modes are saved as complex objects. - object_phase: only the first mode of the object phase is saved (as float16).

This is the format which saves the most space- object: save only the complex object

  • probe: save only the complex probe

Default: “object_probe”

--remove_obj_phase_ramp

Use this option to save the final object after removing the phase ramp estimated from the imperfect centring of the diffraction data (sub-pixel shift). Calculated diffraction patterns using such a corrected object will present a sub-pixel shift relative to the diffraction data.

Default: False

--save

Possible choices: final, all

When to save object and probe. Using “all” will save after each comma-separated algorithm step.

Default: “final”

--data2cxi

Possible choices: crop

Option to save the raw data in CXI format (http://cxidb.org/cxi.html), with all the required information for a ptychography experiment (energy, detector distance, scan number, translation axis are all required). if data2cxi=crop is used, the data will be saved after centering and cropping (default is to save the raw data). If this keyword is present, the processing stops after exporting the data.

Default: False

--movie

create a movie of the scan with all scan positions and diffraction frames. Requires matplotlib and ffmpeg.

Initial object and probe#

--obj_max_pix, --objmaxpix

maximum size in pixels for the object. This is mostly to avoid mistakes when e.g. scan coordinates are incorrectly scaled (*1e6) and result in absurd dimensions.

Default: 8000

--obj_margin, --objmargin

margin (in pixels) around the calculated object area. this is useful when refining positions, to avoid getting outside the object area.

Default: 32

--load

load object and probe from previous optimization, giving a pathto a .cxi or .npz result file. Note that theobject and probe will be scaled if the number of pixels is different for the probe..

--load_probe, --loadprobe

load only probe from previous optimization, giving a pathto a .cxi or .npz result file. Note that the probe will be scaled if the number of pixels is different for the probe..

--load_pixel_size, --loadpixelsize, --load_pixelsize

Pixel size (in meters) from a loaded probe (and possibly object). If the pixel size is different, the loaded arrays will be scaled to match the new pixel size. [default: when loading previous files, object/probe pixel size is calculated from the size of the probe array, assuming same detector distance and pixel size]

--probe

String defining the initial probe, mandatory if –load or –loadprobe is not used. Examples:

  • focus,60e-6x200e-6,0.09: slits size (horizontal x vertical), focal distance (all in meters)

  • focus,200e-6,0.09: radius of the circular aperture, focal distance (all in meters)

  • gaussian,100e-9x200e-9: gaussian type with horizontal x vertical FWHM, both given in meters

  • disc,100e-9: disc-shape, with diameter given in meters

--defocus

defocus distance (+: towards detector). The initial probe is propagated by this distance before being used. This is true both for calculated probes (using probe=…) and for probes loaded from a previous file

--rotate

rotate the initial probe (either simulated or loaded) by X degrees

--object

String to define the original object. It will be initialisedover the entire area using random values. The four given values correspond to the amplitude min and max and the phase min and max (in radians).

Default: “random,0.95,1,0,0.1”

--multiscan_reuse_ptycho, --multi_scan_reuse_ptycho, --multiscanreuseptycho
if used as a keyword, successive scans will re-start from the previous ptycho object and probe, and skip initialisation steps. Useful for ptycho-tomo.This can also be used to supply a shorter algorithm chain which will be used after the firstscan, e.g. with:

--algorithm ML**100,AP**100,DM**1000,nbprobe=2,probe=1--multiscan_reuse_ptycho ML**100,AP**100,probe=1,AP**50,probe=0

In the above example, the probe would be re-used so there is no need to use nbprobe=2 ,and the first step (AP**50) would only optimise the object. Note that if you want to re-process the first scan with this short algorithm chain,it is possible to list the first scan twice, e.g.: scan=12,12,13,14,15.[By default, every scan starts from a new object and probe]

MPI options#

--mpi

Possible choices: split, scan, multiscan

mpi option (only when launching the script using mpiexec): either distribute the list of scans to different processes (scan or multiscan, the default), or split a large scan in different parts, which are automatically aligned andstitched. Examples:

  • mpiexec -n 2 pynx-ptycho-cxi scan=11,12,13,14 data=scan%02d.cxi mpi=multiscan probe=focus,120e-6x120e-6,0.1 defocus=100e-6 verbose=50 algorithm=analysis,ML**100,DM**200,nbprobe=2,probe=1

  • mpiexec -n 4 pynx-ptycho-cxi  mpi=split data=scan151.cxi verbose=50 probe=focus,120e-6x120e-6,0.1 defocus=100e-6 algorithm=analysis,ML**100,DM**200,nbprobe=2,probe=1

Default: “scan”

--mpi_split_nb_overlap

depth of shared neighbours between overlapping regions when using “–mpi split”

Default: 1

--mpi_split_nb_neighbour

target number of shared neighbours for each set for overlap when using “–mpi split”

Default: 20

GPU#

--gpu

string matching GPU name (or part of it, case-insensitive)

--stack_size, --stacksize

number of frames processed together. This is normally set automatically to a large value for better GPU performance. Smaller values than the number of frames can be used to avoid save memory during ML. An optimal value is such that (nb_frame % stack_size) is as small as possible.

Miscellaneous#

--instrument

instrument name (mostly for CXI output).

Default: “ESRF id16a”

--share_probe, --share-probe

Possible choices: -1, 0, 1, 2, 3, 4, 5, 6

Use this option to process multiple scans with the same probe, which will be shared periodically using a tailored algorithm chain, e.g.:

--algorithm "LoopScan(ML**50)**20,LoopScan(DM**50)**40,probe=1"

will perform 2000 DM followed 1000 ML cycles, averaging the probe every 50 cycles. By default the probe will be shared while the optimisation loops over the different scans, but by giving a integer value n, the probe can instead be averaged at the end of each loop (value=1) or decomposed (using truncated SVD) if n>1. [EXPERIMENTAL]

ID16A (near field) parameters#

--data

path to the bliss file, e.g. path/to/data.h5.This can be a pattern, e.g. –data ‘sample1_%04d.h5’, in which case the field will be replaced by the scan number.

--meta

path to the bliss metadata file, e.g. path/to/meta.h5.

--data_ref

path to hdf5 file with reference images (a.k.a. ‘empty beam’)of the direct beam, which are needed for Paganin and CTF algorithms, e.g. ‘–data_ref path/to/ref.h5’

--ptycho_motors, --ptychomotors

name of the two motors used for ptychography, optionally followed by a mathematical expression to be used to calculate the actual motor positions (axis convention, angle..) in meters. Values will be read from the dat files:

  • --ptychomotors=spy,spz,-x*1e-6,y*1e-6 (the default)

  • --ptychomotors pix piz

Note that if the --xy=-y,x command-line argument is used, it is applied _after_ this, using --ptychomotors=spy,spz,-x,y is equivalent to --ptychomotors spy spz --xy=-x,y

Default: [‘spy’, ‘spz’, ‘-x*1e-6’, ‘y*1e-6’]

--delta_beta

delta/beta value, required for Paganin or CTF algorithms.Can also be set in the algorithm string

--adu_scale

optional scale factor by which the all data (including data_ref and dark) will be multiplied in order to have photon counts.

--tomo_motor

Name of the tomography motor -only used to export the rotation angle in output files for ptycho-tomo.

Default: “somega”

--padding

padding value [DEPRECATED, does not work]

Default: 0

--save_plot, --saveplot

Possible choices: object_phase, object_rgba

will save plot at the end of the optimization (png file). A string can be given to specify if only the object phase (default if only –saveplot is given) or rgba should be plotted.

Default: False

Examples:

  • pynx-ptycho-id16a-nf --data data.h5 --meta meta.h5 --dark dark000.h5--algorithm analysis,ML**100,DM**200,nbprobe=3,probe=1

Using a reference (direct beam) frame, allows to use Paganin, and with use_direct_beam will provide an absolute scale for the probe. Also, only save the object phase for a smaller output:

  • pynx-ptycho-id16a-nf --data data.h5 --meta meta.h5 --dark dark000.h5 --data_ref ref.h5--algorithm analysis,ML**100,DM**200,Paganin,probe=1--use_direct_beam --cxi_output object_phase

On this page
Show Source