Scripts Reference

Command-line scripts are available, both Ptychography and CDI scripts.

Ptychography

These are instructions to run the command-line scripts such as pynx-cxipty.py, pynx-id01pty.py, pynx-id13pty.py and pynx-id16apty.py.

This help text can be simply obtained by typing the script without any parameter.

Generic instructions

Here are generic instructions applying to all the scripts:

generic (not beamline-specific) command-line arguments: (all keywords are case-insensitive)
    scan=56: scan number (e.g. in specfile) [mandatory, unless cxifile is used as input].
             Alternatively a list or range of scans can be given:
                scan=12,23,45 or scan="range(12,25)" (note the necessary quotes when using range)

    maxframe=128: limit the total number of frames used to the first N
                  [default=None, all frames are used]

    maxsize=256: the frames are automatically cropped to the largest possible size while keeping 
                 the center of gravity of diffraction in the frame center. Use this to limit the 
                 maximum frame size, for tests or limiting memory use [default=512]

    moduloframe=n1,n2: instead of using all sequential frames of the scan, only take one in n1.
                       if n2<n1 is also given, then the frame numbers taken will be the n for which
                       n % n1 == n2. This can be used to perform two reconstructions with half of 
                       the frames, and then analyse the resolution using Fourier ring correlation.
                       If both moduloframe and maxframe are given, the total number of frames taken 
                       is still maxframe.
                       [default: take all frames]

    algorithm=ML**50,DM**100,probe=1: algorithm for the optimization: [default='ML**50,DM**100']
    
        The algorithm used is:
        - divided in 'steps' separated by commas, e.g: ML**50,DM**100
        - interpreted from right to left, as a mathematical operator to an object on the 
          right-hand side
        - should not contain any space, unless it is given between quotes ('')
        
        - the first type of commands can change basic parameters or perform some analysis:
          - probe=1 or 0: activate or deactivate the probe optimisation (by default only the 
            object is optimised)
          - object=1 or 0: activate or deactivate the object optimisation
          - background=1 or 0: activate or deactivate the background optimisation 
            (only works with AP)
          - position=1 or 0: activate or deactivate position optimisation (for AP or ML, needs some prior convergence)
          - nbprobe=3: change the number of modes for the probe (can go up or down)
          - regularization=1e-4: setting the regularization parameter for the object, to penalize 
                                 local variations in ML runs and smooth the solution
          - obj_smooth=1.5 or probe_smooth=1.0: these parameters will partially smooth the object and probe inertia,
            softening the resulting arrays. This applies to DM and AP algorithms.
          - obj_inertia=0.01 or obj_probe=0.001: these parameters set the inertia of the object and/or probe update,
            yielding more stable result. This applies to DM and AP algorithms.
          - ortho: will perform orthogonalisation of the probe modes. The modes are sorted by 
            decreasing intensity.
          - analysis: perform an analysis of the probe (propagation, modes). Useful combined 
            with 'saveplot' to save the analysis plots
        
        - the second type are operators which will be applied to the Ptycho object:
          - AP: alternate projections. Slow but converging algorithm
          - DM: difference map. Fast early convergence, oscillating after.
          - ML: maximum likelihood conjugate gradient (Poisson-noise). Robust, converging,
            for final optimization.
          These operators can be combined mathematically, e.g.:
          - DM**100: corresponds to 100 cycles of difference map
          - ML**40*DM**100: 100 cycles of DM followed by 40 cycles of ML (note the order)


        Example algorithms chains:
          - algorithm=ML**40,DM**100,probe=1: activate probe optimisation, 
            then 100 DM and 40 ML (quick)
          - algorithm=ML**100,DM**200,nbprobe=3,ML**40,DM**100,probe=1,DM**100: first DM with 
            object update only,  then 100 DM also updating the probe, then use 3 probe modes 
            and do 100 DM followed by 40 ML
          - algorithm=ML**100*AP**200*DM**200,probe=1: 200 DM then 200 AP then 100 ML (one step)
          - algorithm='(ML**10*DM**20)**5,probe=1': 
            repeat 5 times [20 cycles of DM followed by 5 cycles of ML]
            (note the quotes necessary for the parenthesis)

    nbrun=10: number of optimizations to perform [default=1]

    run0=10: number for the first run (can be used to overwrite previous run results) 
             [default: after previous results or 1]

    liveplot: liveplot during optimization [default: no display]

    saveplot: will save plot at the end of the optimization (png file) [default= not saved].
              Optionally this can also specify if only the object phase should be plotted, e.g.:
              saveplot=object_phase: will display the object phase
              saveplot=object_rgba: will use RGBA to display both amplitude and phase.

    saveprefix=ResultsScan%04d/Run%04d: prefix to save the optimized object and probe 
              (as a .cxi or .npz file) and optionally image (png) [default='ResultsScan%04d/Run%04d']

    output_format='cxi': choose the output format for the final object and support.
                         Possible choices: 'cxi', 'npz'
                         [Default='cxi']

    save=all: either 'final' or 'all' this keyword will activate saving after each optimization 
              step (ptycho, ML) of the algorithm in any given run [default=final]

    load=Results0057/Run0001.cxi (or .npz): load object and probe from previous optimization. Note that the
               object and probe will be scaled if the number of pixels is different for the probe.
              [default: start from a random object, simulate probe]


    loadprobe=Results0057/Run0001.npz (or .cxi): load only probe from previous optimization 
                                       [default: simulate probe]

    loadpixelsize=8.6e-9: specify the pixel size (in meters) from a loaded probe 
                          (and possibly object). If the pixel size is different,
                          the loaded arrays will be scaled to match the new pixel size.
                          [default: when loading previous files, object/probe pixel size is 
                          calculated from the size of the probe array, assuming same detector 
                          distance and pixel size]

    probe=focus,60e-6x200e-6,0.09: define the starting probe, either using:
                                  focus,60e-6x200e-6,0.09: slits size (horizontal x vertical),
                                                           focal distance (all in meters)
                                  focus,200e-6,0.09: radius of the circular aperture, 
                                                     focal distance (all in meters)
                                  gaussian,100e-9x200e-9: gaussian type with horizontal x vertical
                                                          FWHM, both given in meters
                                  disc,100e-9: disc-shape, with diameter given in meters
                                  [mandatory, ignored if 'load' or 'loadprobe' is used]

    defocus=1e-6: defocused position (+: towards detector). The probe is propagated by this distance
                  before being used. This is true both for calculated probes (using 'probe=...') 
                  and for probes loaded from a previous file.

    rotate=30: rotate the probe (either simulated or loaded) by X degrees [default: no rotation]

    object=random,0.9,1,0,6: specify the original object values. The object will be initialised
                             over the entire area using random values: random,0-1,0-6.28 : random 
                             amplitudes between 0 and 1, random phases between 0 and 6.28.
                             For high energy small-angle ptycho (i.e. high transmission), 
                             recommended value is: random,0.9,1,0,0
                             [default: random,0,1,0,6.28]

    verbose=20: print evolution of llk (and display plot if 'liveplot' is set) every N cycle 
                [default=50]

    data2cxi: if set, the raw data will be saved in CXI format (http://cxidb.org/cxi.html), 
              will all the required information for a ptychography experiment (energy, detector 
              distance, scan number, translation axis are all required). if 'data2cxi=crop' 
              is used, the data will be saved after centering and cropping (default is to save 
              the raw data). If this keyword is present, the processing stops after exporting the data.

    mask= or loadmask=mask.npy: the mask to be used for detector data, which should have the same 2D
                       shape as the raw detector data.
                       This should be a boolean or integer array with good pixels=0 and bad ones>0
                       (values are expected to follow the CXI convention)
                       Acceptable formats:
                       - mask.npy, mask.npz (the first data array will be used)
                       - mask.edf or mask.edf.gz (a single 2D array is expected)
                       - "mask.h5:/entry_1/path/to/mask" hdf5 format with the full path to the 
                         2D array. 'hdf5' is also accepted as extension.
                       - 'maxipix': if this special name is entered, the masked pixels will be rows 
                         and columns multiples of 258+/-3

    roi=xmin,xmax,ymin,ymax: the region-of-interest to be used for actual inversion. The area is taken 
                             with python conventions, i.e. pixels with indices xmin<= x < xmax and 
                             ymin<= y < ymax.
                             Additionally, the shape of the area must be square, and 
                             n=xmax-xmin=ymax-ymin must also be a suitable integer number
                             for OpenCL or CUDA FFT, i.e. it must be a multiple of 2 and the largest number in
                             its prime factor decomposition must be less or equal to the largest value
                             acceptable by clFFT (<=13 as of November 2016) or cuFFT (<=7).
                             If n does not fulfill these constraints,
                             it will be reduced using the largest possible integer smaller than n.
                             This option supersedes 'maxsize' unless roi='auto'.
                             Other possible values:
                             - 'auto': automatically selects the roi from the center of mass 
                                       and the maximum possible size. [default]
                             - 'all' or 'full': use the entire, uncentered frames. Only useful for pre-processed
                                      data. Cropping may still be performed to get a square and 
                                      FFT-friendly size.

    rebin=2: the experimental images can be rebinned (i.e. a group of n x n pixels is replaced by a
             single one whose intensity is equal to the sum of all the pixels). This 'rebin' is 
             performed last: the ROI, mask, background, pixel size should all correspond to 
             full (non-rebinned) frames. 
             [default: no rebin]

    autocenter=0: by default, the object and probe are re-centered automatically after each 
                  optimisation step, to avoid drifts. This can be used to deactivate this behaviour
                  [default=True]
    
    center_probe_n, center_probe_max_shift: during DM, the probe can shift. The probe center of mass  will be checked
                                            every center_probe_n cycles, and the object and probe position will be
                                            corrected if the center deviates by more than center_probe_max_shift pixels.
                                            This is ignored for near field ptycho.
                                            [default: center_probe_n=5, center_probe_max_shift=5]
    
    dm_loop_obj_probe=3: during DM, when both object and probe are updated, it can be more stable to loop the object
                         and probe update for a more stable optimisation, but slower. [default: 2]
                  
    detector_orientation=1,0,0: three flags which, if True, will do in this order: 
                                array transpose (x/y exchange), flipud, fliplr [default: no change]
                                The changes also apply to the mask.
    
    xy=y,x: order and expression to be used for the XY positions (e.g. '-x,y',...). Mathematical 
            operations can also be used, e.g.: xy=0.5*x+0.732*y,0.732*x-0.5*y
            [default: None- may be superseded in some scripts e.g. for ptypy]
    
    flatfield=flat.npy: the flatfield correction to be applied to the detector data. The array must
                        have the same shape as the frames, which will be multiplied by this 
                        correction.
                        Acceptable formats:
                        - flat.npy, flat.npz (the first data array will be used)
                        - flat.edf or flat.edf.gz (a single 2D array is expected)
                        - "flat.h5:/entry_1/path/to/flat" hdf5 format with the full path to the 
                          2D array. 'hdf5' is also accepted as extension.
                        - flat.mat: from a matlab file. The first array found is loaded
                       [default: no flatfield correction is applied]
    
    dark=dark.npy: the dark correction to be subtracted from the detector data. The array must have 
                       the same shape as the frames, which will be substracted by this correction.
                       Acceptable formats:
                       - dark.npy, dark.npz (the first data array will be used)
                       - dark.edf or dark.edf.gz (a single 2D array is expected)
                       - "dark.h5:/entry_1/path/to/dark" hdf5 format with the full path to the 
                         2D array. 'hdf5' is also accepted as extension.
                       - dark.mat: from a matlab file. The first array found is loaded
                       [default: no dark correction is applied]
    
    orientation_round_robin: will test all possible combinations of xy and detector_orientation 
                             to find the correct detector configuration.

Specific instructions

Instructions for beamline-dedicated scripts can be fully obtained by executing the script without any parameter.

pynx-cxipty.py:

Script to perform a ptychography analysis on data recorded in CXI format (http://cxidb.org/cxi.html)

Examples:
    pynx-cxipty.py data=data.cxi probe=focus,60e-6x200e-6,0.09 
      algorithm=analysis,ML**100,DM**200,nbprobe=3,probe=1 verbose=10 save=all saveplot liveplot

command-line arguments:
    cxifile=/some/dir/to/data.cxi or data=some/dir/to/data.cxi: path to CXI data [mandatory]

    scan=56: scan number [optional: this is used to name the output directory, otherwise 0].
             Several scans can also processed by giving a generic cxi name and scan numbers, 
             e.g.: data=data%03d.cx scan=13,45,78

    gpu=Titan: GPU name for OpenCL calculation 
               [default = will be auto-selected from available GPUs]
    
    xyrange=-2e-6,2e-6,-5e-6,-2e-6: range where the data points will be taken into 
             account (xmin, xmax, ymin, ymax). All scan positions outside this range are ignored.
             This must be given in orginal coordinates, in meters.
             [default=No spatial restrictions]

pynx-id01pty.py:

Script to perform a ptychography analysis on data from id01@ESRF

Example:
    pynx-id01pty.py specfile=siemens.spec scan=57 detectordistance=1.3 ptychomotors=pix,piz,-x,y
      probe=60e-6x200e-6,0.09 algorithm=analysis,ML**100,DM**200,nbprobe=3,probe=1 
      loadmask=maxipix verbose=10 save=all saveplot liveplot

command-line arguments:
    specfile=/some/dir/to/specfile.spec: path to specfile [mandatory]

    scan=56: scan number in specfile [mandatory].
             Alternatively a list or range of scans can be given:
                scan=12,23,45 or scan="range(12,25)" (note the quotes)

    imgcounter=mpx4inr: spec counter name for image number 
                        [default=auto, will use mpx4inr or ei2minr]

    imgname=/dir/to/images/prefix%05d.edf.gz: images location with mask 
        [default: will be extracted from ULIMA_mpx4 or ULIMA_eiger2M the spec scan header]

    nrj=8: energy in keV [default: will be extracted from UMONO spec scan header,  mononrj=x.xxkeV]

    detectordistance=1.3: detector distance in meters 
                          [mandatory, unless given in spec header UDETCALIB]

    pixelsize=55e-6: pixel size on detector in meters [default:55e-6]

    ptychomotors=pix,piz,-x,y: name of the two motors used for ptychography, optionally followed 
                               by a mathematical expression to be used to calculate the actual
                               motor positions (axis convention, angle..). Values will be extracted
                               from the spec files, and are assumed to be in microns.
                               Example 1: ptychomotors=pix,piz
                               Example 2: ptychomotors=pix,piz,-x,y
                               Note that if the 'xy=-y,x' command-line argument is used, 
                               it is applied after this, using 'ptychomotors=pix,piz,-x,y' 
                               is equivalent to 'ptychomotors=pix,piz xy=-x,y'
                               [Mandatory]

    monitor=opt2: spec name for the monitor counter. The frames will be normalized by the ratio
                  of the counter value divided by the median value of the counter over the entire
                  scan (so as to remain close to Poisson statistics). A monitor intensity lower 
                  than 10% of the median value will be interpreted as an image taken without beam
                  and will be skipped.
                  [default = None]

    kmapfile=detector/kmap/kmap_00000.edf.gz: if images are saved in a multiframe data file 
                                              This superseeds imgname=...

    gpu=Titan: GPU name for OpenCL calculation [default = Titan]
    
    livescan: if set, this keyword will trigger the analysis of all scans of a given type 
              (by default 'spiralscan') in the supplied spec file. If scan=NN is also given,
              the analysis will start after scan #NN.
              Once all scans are processed, the script will wait for new scans to arrive.

pynx-id13pty.py:

Script to perform a ptychography analysis on data from id13@ESRF

Example:
    pynx-id13pty.py h5file=pty_41_61_master.h5 specfile=scan.dat scan=41 detectordistance=1.3 
      ptychomotors=pix,piz,-x,y probe=60e-6x200e-6,0.09 
      algorithm=analysis,ML**100,DM**200,nbprobe=3,probe=1 verbose=10 save=all saveplot liveplot

Command-line arguments (beamline-specific):
    specfile=/some/dir/to/specfile.spec: path to specfile [mandatory]

    scan=56: scan number in specfile [mandatory].
             Alternatively a list or range of scans can be given:
                scan=12,23,45 or scan="range(12,25)" (note the quotes)

    h5data=entry/data/data_%6d: generic hdf5 path to stack of 2d images inside hdf5 file 
                                [default=entry/data/data_%06d]

    nrj=8: energy in keV [default: will be extracted from the HDF% file,
                          entry/instrument/beam/incident_wavelength]

    detectordistance=1.3: detector distance in meters [mandatory]

    pixelsize=75e-6: pixel size on detectorin meters 
                     [default: will read from hdf5 data: entry/instrument/detector/x_pixel_size]

    ptychomotors=nnp2,nnp3,-x,y: name of the two motors used for ptychography, optionally followed
                                 by a mathematical expression to be used to calculate the actual 
                                 motor positions (axis convention, angle..). Values will be 
                                 extracted from the edf files, and are assumed to be in microns.
                                 Example 1: ptychomotors=nnp2,nnp3
                                 Example 2: ptychomotors=nnp2,nnp3,-x,y
                                 Note that if the 'xy=-y,x' command-line argument is used, it is 
                                 applied after this, using 'ptychomotors=nnp2,nnp3,-x,y' is 
                                 equivalent to 'ptychomotors=nnp2,nnp3 xy=-x,y'
                                 [Mandatory]

    monitor=opt2: spec name for the monitor counter. The frames will be normalised by the ratio of
                  the counter value divided by the median value of the counter over the entire scan
                  (so as to remain close to Poisson statistics). A monitor intensity lower than 10%
                  of the median value will be interpreted as an image taken without beam and will be 
                  skipped.
                  [default = None]

    kmapfile=detector/kmap/kmap_00000.edf.gz: if images are saved in a multiframe data file.
                                              This superseeds imgname=...

    gpu=Titan: GPU name for OpenCL calculation [default = Titan]

pynx-id16apty.py:

Script to perform a ptychography analysis on data from id16(a)@ESRF

Example:
    pynx-id16pty.py h5meta=data.h5 probe=60e-6x60e-6,0.09
      algorithm=analysis,ML**100,DM**200,nbprobe=3,probe=1

Command-line arguments (beamline-specific):
    h5meta=data.h5: hdf5 filename with the scan metadata. The scan number, energy, 
                    detector distance will be extracted 
                    [mandatory - if a single *.h5 file is in the working directory,
                     it will be used by default]

    pixelsize=55e-6: pixel size on detector in meters
                     [default: 55e-6 or will be read frommetadata or hdf5 data]

    imgname=/dir/to/images/prefix%05d.edf: images location with mask. Images will be read starting
                                           from 0 or 1, until no more frames are found, or maxframe
                                           has been reached.
                                           To process several scans, several % fields can be added, the last
                                           will be interpreted as the image number and the others replaced by the
                                           scan number. 
                                           Examples:
                                              /dir/to/scan_%04d/prefix_scan%04d_img%05d.edf 
                                              /dir/to/scan_%04d/prefix__img%05d.edf 
                                              /dir/to/images/prefix_img%05d.edf
                                           [mandatory if h5data is not supplied]
                                           
    
    h5data=/path/to/hdf5/data/file.nxs: path to nexus data file (lambda detector).
                                     If one or several % field are included in the name, they will be replaced by
                                     the scan number.
                                     Examples: 
                                        h5data=/path/to/hdf5/data/scan%d/scan%d.nxs
                                        h5data=/path/to/hdf5/data/scan%d.nxs
                                        h5data=/path/to/hdf5/data/data.nxs
                                     [mandatory unless imgname is given, will default 
                                      to the only *.nxs file in directory]

    ptychomotors=spy,spz,-x,y: name of the two motors used for ptychography, or the motor 
                               positions file, optionally followed by a mathematical expression to
                               be used to calculate the actual motor positions (axis convention, angle..).
                               Values are either:
                                   - extracted from the edf files, and are assumed to be in mm.
                                   - or extracted from the 'mot_pos.txt' file, and assumed to 
                                     be in microns..
                               Example 1: ptychomotors=spy,spz
                               Example 2: ptychomotors=spy,spz,-x,y
                               Example 3: ptychomotors=mot_pos.txt
                               Note that if the 'xy=-y,x' command-line argument is used, it is 
                               applied after this, using 'ptychomotors=spy,spz,-x,y' 
                               is equivalent to 'ptychomotors=spy,spz xy=-x,y'
                               [Mandatory]

    detectordistance=1.3: detector distance in meters [default: will be read from hdf5 metadata]

    gpu=Titan: GPU name for OpenCL calculation [Mandatory]

CDI

These are instructions to run the command-line scripts: pynx-id10cdi.py and pynx-id01cdi.py. They can also be used for generic data not from these beamlines.

This help text can be simply obtained by typing the script without any parameter.

Generic instructions

Here are generic instructions applying to all the scripts:

generic (not beamline-specific) command-line/file parameters arguments: (all case-insensitive)

    data=data.npy: name of the data file including the 3D observed intensity.
                   recognized formats include .npy, .npz (if several arrays are included iobs, 
                   should be named 'data' or 'iobs'), .tif or .tiff 
                   (assumes a multiframe tiff image), or .cxi (hdf5).
                   [mandatory unless another beamline-specific method to import data is used]
                   
    detector_distance=0.7: detector distance in meters
    
    pixel_size_data=55e-6: deprecated, use pixel_size_detector instead

    pixel_size_detector=55e-6: pixel size of the supplied data (detector pixel size)
    
    wavelength=1.5e-10: wavelength in meters
    
    verbose=20: the run will print and optionally plot every 'verbose' cycles
    
    live_plot: if used as keyword (or live_plot=True in a parameters file), a live plot 
               will be shown  every 'verbose' cycle. If an integer number N is given, 
               display every N cycle.
    
    gpu=Titan: name of the gpu to use [optional, by default the fastest available will be used]
    
    auto_center_resize: if used (command-line keyword) or =True, the input data will be centered 
                        and cropped  so that the size of the array is compatible with the (GPU) 
                        FFT library used. If 'roi' is used, centering is based on ROI. 
                        [default=False]
    
    roi=0,120,0,235,0,270: set the region-of-interest to be used, with min,max given along each 
                           dimension in python fashion (min included, max excluded), for the 2 or 3
                           axis. ROI coordinates should be indicated before any rebin is done.
                           Note that if 'auto_center_resize' is used, the ROI may still be shrunk
                           to obtain an array size compatible with the FFT library used. Similarly 
                           it will be shrunk if 'max_size' is used but ROI size is larger.
                           Other example: roi=0,-1,300-356,300+256,500-256,500+256
                           [default=None]
                        
    nb_run=1: number of times to run the optimization [default=1]
    
    nb_run_keep: number of best run results to keep, according to likelihood statistics. This is only useful
                 associated with nb_run [default: keep all run results]
    
    data2cxi: if used as keyword (or data2cxi=True in a parameters file), convert the original 
              data to CXI(HDF5)  format. Will be saved to file 'data.cxi', or if a data file
              has been supplied (e.g. data57.npz), to the same file with extension .cxi.
              
    output_format='cxi': choose the output format for the final object and support.
                         Other possible choice: 'npz', 'none'
                         [Default='cxi']
    
    note='This dataset was measure... Citation: Journal of coherent imaging (2018), 729...':
         Optional text note which will be saved as a note in the output CXI file 
         (and also for data2cxi).
         
    instrument='ESRF-idxx': the name of the beamline/instrument used for data collection
                            [default: depends on the script actually called]
                            
    sample_name='GaN nanowire': optional name for the sample
                         
    mask=zero: mask for the diffraction data. If 'zero', all pixels with iobs <= 0 will be masked.
              If 'negative', all pixels with iobs < 0 will be masked. 
              If 'maxipix', the maxipix gaps will be masked.
              If 'maxipix1', the maxipix gaps will be masked, except for the large pixel one ach side of the gap,
              for which the intensity will be divided by 3.
              If 'maxipix6', the maxipix gaps will be linearly interpolated between the gap large pixels.
              Other possibilities: give a filename for the mask file and  import mask from .npy, .npz, .edf, .mat.
              (the first available array will be used if multiple are present) file.
              Pixels = 0 are valid, > 0 are masked. If the mask is 2D
              and the data 3D, the mask is repeated for all frames along the first dimension (depth).
              Several masks can be combined, separated by a comma, e.g. mask=maxipix,dead.npz
              [default=None, no mask]
    
    free_pixel_mask=result.cxi: pixel mask for free log-likelihood calculation. By default the free pixel mask is
                                randomly initialised, but this can be used to load a mask from a file for
                                consistency between runs. It can be loaded from a .npy, .edf, .tif, .npz or 
                                .mat file - if .npz or .mat are used, the first array matching the iobs shape is used. 
                                A .cxi filename can also be used as input, in which case it is assumed that this
                                is the result of a previous optimisation, and that the mask can be loaded from
                                '/entry_last/image_1/process_1/configuration/free_pixel_mask'
    
    iobs_saturation=1e6: saturation value for the observed intensity. Any pixel above this intensity will be masked
                         [default: no saturation value]
    
    zero_mask: by default masked pixels are free and keep the calculated intensities during HIO, RAAR, ER and CF cycles.
               Setting this flag will force all masked pixels to zero intensity. This can be more stable with a large 
               number of masked pixels and low intensity diffraction data.
               If a value is supplied the following options can be used:
               zero_mask=0: masked pixels are free and keep the calculated complex amplitudes
               zero_mask=1: masked pixels are set to zero
               zero_mask=auto: this is only meaningful when using a 'standard' algorithm below. The masked pixels will
                               be set to zero during the first 60% of the HIO/RAAR cycles, and will be free during the 
                               last 40% and ER, ML ones.

    object=obj.npy: starting object. Import object from .npy, .npz, .mat (the first available array 
          will  be used if multiple are present), or CXI file.
          [default=None, object will be defined as random values inside the support area]
    
    support=sup.npy: starting support. Import support from .npy, .npz, .edf, .mat (the first 
              available array will be used if multiple are present) file.  Pixels > 0 are in
              the support, 0 outside. if 'auto', support will be estimated using the intensity
              auto-correlation. If 'circle' or 'square', the suppport will be initialized to a 
              circle (sphere in 3d), or a square (cube). Ignored if support_formula is given.
              [default='auto', support will be defined otherwise]
    
    support_size=50: size (radius or half-size) for the initial support, to be used in 
                     combination with 'support_type'. The size is given in pixel units.
                     Alternatively one value can be given for each dimension, i.e. 
                     support_size=20,40 for 2D data, and support_size=20,40,60 for 3D data. 
                     This will result in an initial support which is a rectangle/parallelepiped
                     or ellipsis/ellipsoid.  Ignored if support_formula is given.
                     [if not given, this will trigger the use of auto-correlation 
                      to estimate the initial support]

    support_formula: formula to compute the support. This should be an equation using ix,iy,iz and ir pixel coordinates
                     (all centered on the array) which can be evaluated to produce an initial support (1/True inside, 
                     0/False outside). Mathematical functions should use the np. prefix (np.sqrt, ..). Examples:
                     support_formula="ir<100": sphere or circle of radius 100 pixels
                     support_formula="(np.sqrt(ix**2+iy**2)<50)*(np.abs(iz)<100)": cylinder of radius 50 and height 200

    support_autocorrelation_threshold=0.1: if no support is given, it will be estimated 
                                           from the intensity autocorrelation, with this relative 
                                           threshold.
                                           [default value: 0.1]

    support_threshold=0.25: threshold for the support update. Alternatively two values can be given, and the threshold
                            will be randomly chosen in the interval given by two values: support_threshold=0.20,0.28.
                            This is mostly useful in combination with nb_run.
                            [default=0.25]
    
    support_threshold_method=max: method used to determine the absolute threshold for the 
                                  support update. Either:'max' or 'average' or 'rms' (the default) values,
                                  taken over the support area/volume, after smoothing.
                                  Note that 'rms' and 'average' use the modulus or root-mean-square value normalised 
                                  over the support area, so when the support shrinks, the threshold tends to increase.
                                  In contrast, the 'max' value tends to diminish as the optimisation
                                  progresses. rms is varying more slowly than average and is thus more stable.
                                  In practice, using rms or average with a too low threshold can lead to the divergence 
                                  of the support (too large), if the initial support 

    support_only_shrink: if set or support_only_shrink=True, the support will only shrink 
                         (default: the support can grow again)
                         
    support_smooth_width_begin=2
    support_smooth_width_end=0.25: during support update, the object amplitude is convoluted by a
                                   gaussian with a size
                                   (sigma) exponentially decreasing from support_smooth_width_begin
                                   to support_smooth_width_end from the first to the last RAAR or 
                                   HIO cycle.
                                   [default values: 2 and 0.5]
    
    support_smooth_width_relax_n: the number of cycles over which the support smooth width will
                                  exponentially decrease from support_smooth_width_begin to 
                                  support_smooth_width_end, and then stay constant. 
                                  This is ignored if nb_hio, nb_raar, nb_er are used, 
                                  and the number of cycles used
                                  is the total number of HIO+RAAR cycles [default:500]
    
    support_post_expand=1: after the support has been updated using a threshold,  it can be shrunk 
                           or expanded by a few pixels, either one or multiple times, e.g. in order
                           to 'clean' the support:
                           - support_post_expand=1 will expand the support by 1 pixel
                           - support_post_expand=-1 will shrink the support by 1 pixel
                           - support_post_expand=-1,1 will shrink and then expand the support 
                             by 1 pixel
                           - support_post_expand=-2,3 will shrink and then expand the support 
                             by 2 and 3 pixels
                           - support_post_expand=2,-4,2 will expand/shrink/expand the support 
                             by 2, 4 and 2 pixels
                           - etc..
                           [default=None, no shrinking or expansion]

    support_update_border_n: if > 0, the only pixels affected by the support updated lie within +/- N pixels around the
                             outer border of the support.

    support_update_period=50: during RAAR/HIO, update support every N cycles.
                              If 0, support is never updated.

    positivity: if set or positivity=True, the algorithms will be biased towards a real, positive
                object. Object is still complex-valued, but random start will begin with real 
                values. [default=False]
    
    beta=0.9: beta value for the HIO/RAAR algorithm [default=0.9]
    
    crop_output=0: if >0 (default:4), the output data will be cropped around the final
                   support plus 'crop_output' pixels. If 0, no cropping is performed.
    
    rebin=2: the experimental data can be rebinned (i.e. a group of n x n (x n) pixels is
             replaced by a single one whose intensity is equal to the sum of all the pixels).
             Both iobs and mask (if any) will be rebinned, but the support (if any) should
             correspond to the new size. The supplied pixel_size_detector should correspond
             to the original size. The rebin factor can also be supplied as one value per
             dimension, e.g. "rebin=4,1,2".
             [default: no rebin]
    
    max_size=256: maximum size for the array used for analysis, along all dimensions. The data
                  will be cropped to this value after centering. [default: no maximum size]
                  
    user_config*=*: this can be used to store a custom configuration parameter which will be ignored by the 
                    algorithm, but will be stored among configuration parameters in the CXI file (data and output).
                    e.g.: user_config_temperature=268K  user_config_comment="Vibrations during measurement" etc...

    ############# ALGORITHMS: standard version, using RAAR, then HIO, then ER and ML

    nb_raar=600: number of relaxed averaged alternating reflections cycles, which the 
                 algorithm will use first. During RAAR and HIO, the support is updated regularly

    nb_hio=0: number of hybrid input/output cycles, which the algorithm will use after RAAR. 
                During RAAR and HIO, the support is updated regularly

    nb_er=200: number of error reduction cycles, performed after HIO, without support update

    nb_ml=20: number of maximum-likelihood conjugate gradient to perform after ER

    detwin: if set (command-line) or if detwin=True (parameters file), 10 cycles will be performed
            at 25% of the total number of RAAR or HIO cycles, with a support cut in half to bias
            towards one twin image



    ############# ALGORITHMS: customized version 
    
    algorithm="ER**50,(Sup*ER**5*HIO**50)**10": give a specific sequence of algorithms and/or 
              parameters to be 
              used for the optimisation (note: this string is case-insensitive).
              Important: 
              1) when supplied from the command line, there should be NO SPACE in the expression !
              And if there are parenthesis in the expression, quotes are required around the 
              algorithm string
              2) the string and operators are applied from right to left

              Valid changes of individual parameters include (see detailed description above):
                positivity = 0 or 1
                support_only_shrink = 0 or 1
                beta = 0.7
                live_plot = 0 (no display) or an integer number N to trigger plotting every N cycle
                support_update_period = 0 (no update) or a positive integer number
                support_smooth_width_begin = 2.0
                support_smooth_width_end = 0.5
                support_smooth_width_relax_n = 500
                support_threshold = 0.25
                support_threshold_mult = 0.7  (multiply the threshold by 0.7à
                support_threshold_method=max or average or rms
                support_post_expand=-1#2 (in this case the commas are replaced by # for parsing)
                zero_mask = 0 or 1
                verbose=20
                fig_num=1: change the figure number for plotting
                
              Valid basic operators include:
                ER: Error Reduction
                HIO: Hybrid Input/Output
                RAAR: Relaxed Averaged Alternating Reflections
                DetwinHIO: HIO with a half-support (along first dimension)
                DetwinHIO1: HIO with a half-support (along second dimension)
                DetwinHIO2: HIO with a half-support (along third dimension)
                DetwinRAAR: RAAR with a half-support (along first dimension)
                DetwinRAAR1: RAAR with a half-support (along second dimension)
                DetwinRAAR2: RAAR with a half-support (along third dimension)
                CF: Charge Flipping
                ML: Maximum Likelihood conjugate gradient (incompatible with partial coherence PSF)
                PSF or EstimatePSF: calculate partial coherence point-spread function 
                                    with 50 cycles of Richardson-Lucy
                Sup or SupportUpdate: update the support according to the support_* parameters
                ShowCDI: display of the object and calculated/observed intensity. This can be used
                         to trigger this plot at specific steps, instead of regularly using 
                         live_plot=N. This is thus best used using live_plot=0
              
              Examples of algorithm strings, where steps are separated with commas (and NO SPACE!),
              and are applied from right to left. Operations in a given step will be applied
              mathematically, also from right to left, and **N means repeating N tymes (N cycles) 
              the  operation on the left of the exponent:
                algorithm=HIO : single HIO cycle
                
                algorithm=ER**100 : 100 cycles of HIO
                
                algorithm=ER**50,HIO**100 : 100 cycles of HIO, followed by 50 cycles of ER
                
                algorithm=ER**50*HIO**100 : 100 cycles of HIO, followed by 50 cycles of ER
                
                algorithm="ER**50,(Sup*ER**5*HIO**50)**10" : 
                    10 times [50 HIO + 5 ER + Support update], followed by 50 ER
                
                algorithm="ER**50,verbose=1,(Sup*ER**5*HIO**50)**10,verbose=100,HIO**100":
                    change the periodicity of verbose output
                
                algorithm="ER**50,(Sup*ER**5*HIO**50)**10,support_post_expand=1,
                           (Sup*ER**5*HIO**50)**10,support_post_expand=-1#2,HIO**100"
                    same but change the post-expand (wrap) method
                
                algorithm="ER**50,(Sup*PSF*ER**5*HIO**50)**5,(Sup*ER**5*HIO**50)**10,HIO**100"
                    activate partial correlation after a first series of algorithms
                
                algorithm="ER**50,(Sup*PSF*HIO**50)**4,(Sup*HIO**50)**8"
                    typical algorithm steps with partial coherence
                
                algorithm="ER**50,(Sup*HIO**50)**4,(Sup*HIO**50)**4,positivity=0,
                          (Sup*HIO**50)**8,positivity=1"
                    same as previous but starting with positivity constraint, removed at the end.

                

            [default: use nb_raar, nb_hio, nb_er and nb_ml to perform the sequence of algorithms]     

    save=all: either 'final' or 'all' this keyword will activate saving after each optimisation 
              step (comma-separated) of the algorithm in any given run [default=final]

Specific instructions

Instructions for beamline-dedicated scripts can be fully obtained by executing the script without any parameter.

pynx-id10cdi.py:

Script to perform a CDI reconstruction of data from id10@ESRF.
command-line/file parameters arguments: (all keywords are case-insensitive):

    Specific defaults for this script:
        auto_center_resize=True
        detwin = False
        mask = zero  # use 'mask=no' from the command-line to avoid using a mask
        nb_raar = 600
        nb_hio = 0
        nb_er = 200
        nb_ml = 0
        support_size = 50
        support_type = circle
        zero_mask = 0
        positivity = True

pynx-id01cdi.py:

Script to perform a CDI reconstruction of data from id01@ESRF.
command-line/file parameters arguments: (all keywords are case-insensitive):

    specfile=/some/dir/to/specfile.spec: path to specfile [mandatory, unless data= is used instead]

    scan=56: scan number in specfile [mandatory].
             Alternatively a list or range of scans can be given:
                scan=12,23,45 or scan="range(12,25)" (note the quotes)

    imgcounter=mpx4inr: spec counter name for image number
                        [default='auto', will use either 'mpx4inr' or 'ei2mint']

    imgname=/dir/to/images/prefix%05d.edf.gz: images location with prefix 
            [default: will be extracted from the ULIMA_mpx4 entry in the spec scan header]
            
    Specific defaults for this script:
        auto_center_resize = True
        detwin = True
        nb_raar = 600
        nb_hio = 0
        nb_er = 200
        nb_ml = 0
        support_size = None
        zero_mask = auto