PyNX: Python tools for Nano-structures Xtallography¶
NEWS¶
2020/12/12: PyNX 2020.2.2 is available with a few fixes. See the full Changelog
2020/10/23: PyNX 2020.2 is available, notably introducing distributed computing on multiple GPU
2020/09: the PyNX article: PyNX: high-performance computing toolkit for coherent X-ray imaging based on operators is out: J. Appl. Cryst. 53 (2020), 1404, also available as arXiv:2008.11511
2020/02/02: version 2020.1 is out !
2019/06/19: version 2019.2.6 is out. Note that 2019.2.x (x<6) versions had incorrect Ptycho CUDA scaling, preventing correct ML minimisation
2018/06: all coherent imaging modules have now been converted to use an operator-based API
2016/09: overhaul of the library structure -
pynx.gpu
has now becomepynx.scattering
, withpynx.scattering.gid
as a sub-module.
Introduction¶
PyNX stands for Python tools for Nano-structures Xtallography. It can be used for:
Coherent X-ray imaging simulation and analysis: coherent diffraction imaging (CDI), Ptychography, Wavefront propagation, near field and far field techniques…
Fast scattering calculations from large number of atoms and reciprocal space positions.
PyNX is fully optimised to use Graphical Processing Units, using either CUDA or OpenCL, to provide fast calculations with 1 to 3 orders of magnitude speedup compared to standard processor calculations.
PyNX scripts¶
PyNX can be used simply with command-line scripts for some applications (2D/3D CDI and 2D Ptychography). These can take generic files as input, such as CXI files (http://cxidb.org), or can analyse data directly from beamlines.
PyNX as a python toolkit¶
PyNX can be used as a python library with the following main modules:
pynx.scattering
: X-ray scattering computing using graphical processing units, allowing up to 2.5x10^11 reflections/atoms/seconds (single nVidia Titan X). The sub-module``pynx.scattering.gid`` can be used for Grazing Incidence Diffraction calculations, using the Distorted Wave Born Approximationpynx.ptycho
: simulation and analysis of experiments using the ptychography technique, using GPU (OpenCL). Examples are available in the pynx/Examples directory. Scripts for analysis of raw data from beamlines are also available, as well as using or producing ptychography data sets in CXI (Coherent X-ray Imaging) format.pynx.wavefront
: X-ray wavefront propagation in the near, far field, or continuous (examples available at the end ofwavefront.py
). Also provided are sub-modules for Fresnel propagation and simulation of the illumination from a Fresnel Zone Plate, both using OpenCL for high performance computing.pynx.cdi
: Coherent Diffraction Imaging reconstruction algorithms using GPU for Coherent Diffraction Imaging, in 2D or 3D, for small-angle or Bragg diffraction data. This uses either CUDA or OpenCL, but CUDA is strongly recommended for 3D data (significant speedup).
Download¶
- PyNX is available from:
http://gitlab.esrf.fr/favre/PyNX (login required, site registration is open & free)
Mailing list, git & issue tracker¶
There is a mailing list for PyNX: to subscribe, send an email to pynx-subscribe@esrf.fr
To access the git repository and the issue tracker, you can create an account on http://gitlab.esrf and ask the developers for access to the PyNX project at http://gitlab.esrf.fr/favre/PyNX
Changelog¶
Version 2020.2.2 (2020-12-12)¶
CDI modes analysis: add CUDA version for accelerated phase matching
Ptycho: in the CXI output, enforce using a local link for the object virtual dataset, to avoid a broken link. Workaround for https://github.com/h5py/h5py/issues/1546 in h5py<3.0
Version 2020.2.1 (2020-12-05)¶
CDI runner: correct bug where mask file with uppercase characters was not correctly imported
Add cuFFT workaround for CUDA>=11 (see https://github.com/lebedov/scikit-cuda/issues/308)
Version 2020.2 (2020-10-23)¶
Ptycho-MPI: add ability to distribute computing on several GPU and/or nodes, using MPI. This can be used to distribute independent scans (mpi=multiscan) or to split a large scan (mpi=split) in multiple parts which are automatically stitched (aligned in phase and positions)
Ptycho: systematic and reproducible phase ramp removal (far field)
Ptycho: keep track of original, absolute coordinates in object
Ptycho: use NeXus formatting to automatically display object absolute coordinates
Ptycho simulation script: better scale the probe, step size and object according to the frame size. Allow using a siemens star as an object. Use GPU calculations for simulation.
Ptycho: add a threshold filter for the position correction, to avoid updating shifts where the object gradient is too small.
Ptycho runner: add obj_max_pix parameter for the maximum object size
Ptycho: the absolute position of object & probe is corrected in save plots as well as the CXI file.
Ptycho: rename pynx-analyzeprobe.py to pynx-ptycho-analysis.py, and allow using CXI as input files.
CDI: add interpolation scheme for masked pixels (e.g. in gaps), using inverse distance weighted interpolation, and a large confidence interval for amplitude projection during optimisation.
CDI-MPI: add ability to distribute multiple scans or runs using MPI
CDI & Ptycho runner: access help text using ‘help’ or ‘–help’ keywords
CDI runner: allow processing multiple scans in a more generic way
CDI runner: allow importing the initial support from a previous CXI result file, or from an hdf5 file, optionally with the hdf5 path to the support data array.
CDI: add iobs plot to PRTF plots.
CDI: save the correct un-masked iobs in CXI files
CDI, Ptycho: more accurate reporting of average dt/cycle, including support update, graphical displays.
pynx-cdi-analysis: save PRTF in modes.h5 when possible
Documentation: now include notebooks in the html docs (using nbsphinx)
HDF5: set by default the HDF5_USE_FILE_LOCKING environment variable to FALSE when opening a data file which may also be written by another process simultaneously. This is inhibited if HDF5_USE_FILE_LOCKING is already set.
Version 2020.1 (2020-02-02)¶
CDI runner: enable combining several masks and interpolating gap maxipix masks
CDI runner: enable setting initial support based on command-line equation
CDI: Faster cdi array matching and pynx-cdi-analysis using OpenCL
CDI: add phase retrieval transfer function (PRTF) plotting code
CDI & ptychography: more automatic tests
Ptycho: enable position/translation corrections
[BUG] Ptycho: correct gradient calculation for maximum likelihood/conjugate gradient algorithm
[BUG] Ptycho runner: correctly reshape and rescale probe as needed when loading a previous probe
All: use safe import for matplotlib.pyplot in case tk is not available, switching backend to agg
pynx-test.py: add option for automatic tests & email reporting
More efficient memory usage, especially for tests
[INCOMPATIBLE CHANGE] Scattering: change sign in Fhkl_thread, now computing F(hkl)=SUM_i exp(-2j*pi*(h*x_i + k*y_i + l*z_i)) instead of F(hkl)=SUM_i exp(+2j*pi*(h*x_i + k*y_i + l*z_i))
Version 2019.2.6 (2019-06-19)¶
[BUG] Ptycho: Correct CUDA ML operator, which prevented correct minimisation
CDI: keep the free pixel mask during successive runs (nb_run=N)
Version 2019.2.5 (2019-06-02)¶
Ptycho: large speedup when using CUDA by increasing stack size (needed for fast, recent GPU) and atomic operations
Ptycho: store history of figures of merit and cycle parameters. Export to CXI file.
Ptycho: add pynx-simulationpy runner for tests
Ptycho: add nanoscopium runner script
Ptycho: add dm_loop_obj_probe parameter to control looping over object+probe update
Improve pynx-test.py output
Improve documentation
Improve pynx.test.speed to test for large pinned memory allocations.
[BUG] CDI: fix CDI Calc2Obs operator
[BUG] CDI & Ptycho: correct nps file import
[BUG] Ptycho: correct wavelength calculation for CXI export
[BUG] Ptycho CXI runner: correct xyrange parameter interpretation
Version 2019.2 (2019-05-20)¶
CDI & Ptychography: CXI output files follow the NeXus standard, allowing direct display when opened with silx view.
CDI: record history of indicators (log-likelihood, support size and levels, …) in CXI output
CDI runner: add save=all option to save several steps in the algorithm chain
CDI: support update has been improved to avoid diverging, affecting threshold levels to be used.
CDI: allow updating support only around the border of the support (support_update_border_n)
CDI: add GPS operator
CDI: export a more complete set of configuration parameters to CXI files
CDI: correct scaling (ML operator, initial scaling)
CDI: correct examples
CDI runner: add save=all options to export solved object after each step
Ptychography: CUDA operators are now preferred to OpenCL (significantly faster for large frame sizes)
Ptychography: improve near field algorithm, allowing to specify mask with a zero-phase restraint (vacuum)
Ptycho runner: allow to roll (circular shift) data instead of cropping
Ptycho: add NanoMAX (MaxIV) runner script
Utils: add phase retrieval transfer function estimation (for CDI)
Version 2019.1 (2019-02-07)¶
CDI: add ‘free’ log-likelihood figure-of-merit.
CDI: allow to give a range for the support threshold, when performing multiple runs.
CDI: allow to keep only the best solutions when performing multiple runs
CDI: id01 and id10 scripts will now print the algorithm chain used, when it is not user-supplied
CDI: add pynx-cdi-analysis script to analyse proposed solutions.
Ptycho: enable CUDA operators, 2x speed improvements, especially for large frame sizes
Ptycho: correct probe and object orientation and axis in plots, so that both are seen from the source
Ptycho: auto-correct probe centering when necessary (DM)
Ptycho: better handling of plots for near field Ptycho
Ptycho scripts: add ability to create a movie of the scan
CDI & Ptycho: improved speed of calculations from GPU profiling.
Add test suite
Support for Python 3.7
Python 3.4 is deprecated
[Incompatible change] Ptycho: now all API functions using x,y(,z) coordinates as input or output will use them in alphabetical order. The inverse order is only used for shapes e.g. (ny, nx). This affects notably declaration of PtychoData, as well as get_view_coord(), calc_obj_shape(), Simulation.scan.values
[BUG] CDI: correct handling of smooth parameter in OpenCL SupportUpdate() operator
[BUG] CDI: correct handling of masked pixels when using auto-correlation to init the support (OpenCL)
[BUG] Ptycho: correct taking into account of mask when using a command-lien script
[BUG] Ptycho script: correct taking into account mask in some circumstances
See the full Changelog
Citation & Bibliography¶
If you use PyNX for scientific work, please consider including a citation:
If you use PyNX for coherent X-ray Imaging including CDI and ptychography:
Cite the 2020 PyNX article: J. Appl. Cryst. 53 (2020), 1404–1413
Give a link to the project: http://ftp.esrf.fr/pub/scisoft/PyNX/
If you use PyNX for GPU scattering calculations:
Cite the first PyNX article: J. Appl. Cryst. 44(2011), 635-640. A preprint version is also available on ArXiv:1010.2641
Give a link to the project: http://ftp.esrf.fr/pub/scisoft/PyNX/
- PyNX re-uses or was inspired by features described in the following articles and open-source software packages:
PtyPy: 1. B. Enders and P. Thibault, “A computational framework for ptychographic reconstructions”, Proc Math Phys Eng Sci 472(2196), (2016).
M. Odstrčil, A. Menzel, and M. Guizar-Sicairos, “Iterative least-squares solver for generalized maximum-likelihood ptychography,” Optics Express 26(3), 3108 (2018).
S. Marchesini, A. Schirotzek, C. Yang, H. Wu, and F. Maia, “Augmented projections for ptychographic imaging,” Inverse Problems 29(11), 115009 (2013).
Thibault and A. Menzel, “Reconstructing state mixtures from diffraction measurements,” Nature 494(7435), 68–71 (2013).
Thibault and M. Guizar-Sicairos, “Maximum-likelihood refinement for coherent diffractive imaging,” New J. Phys. 14(6), 063004 (2012).
J. N. Clark, X. Huang, R. Harder, and I. K. Robinson, “High-resolution three-dimensional partially coherent diffraction imaging,” Nat Commun 3, 993 (2012).
S. Marchesini, “A unified evaluation of iterative projection algorithms for phase retrieval,” Review of Scientific Instruments 78(1), 011301 (2007).
License¶
The PyNX library is distributed with a CeCILL-B license (an open-source license similar to the FreeBSD one). See http://www.cecill.info/licences/Licence_CeCILL-B_V1-en.html
Note that CPU computing of the pynx.scattering
module uses the sse_mathfun.h
header, which is distributed
under the zlib license. See http://gruntthepeon.free.fr/ssemath/
See http://ftp.esrf.fr/pub/scisoft/PyNX/README.txt for more details about the license, copyright, as well as other possible issues regarding ptychography.
Installation¶
- Installation
How to install PyNX (preferably using a python virtual environment)
Automated testing¶
To automatically test PyNX after installation, you can run the pynx-test.py script, which will run a series of tests and can help diagnose issues specific to GPU languages (OpenCL, CUDA), dependencies or applications (CDI, Ptycho..). Alternatively you can run pytest (if installed) from the root of the PyNX source directory.
Beginner tutorials¶
To begin using PyNX, you can read the following Tutorials:
Use command-line-scripts for data analysis:
Python API tutorial notebooks for:
Ptychography
Coherent Diffraction Imaging
Wavefront propagation
Scattering calculations
Command-line scripts¶
- Scripts Reference
Documentation of scripts included in PyNX
API Documentation¶
- API Reference
Documentation of modules included in PyNX