PyNX: Python tools for Nano-structures Xtallography

NEWS

  • 2021/11/XX: PyNX 2021.1 is available. See the full Changelog

  • 2020/12/12: PyNX 2020.2.2 is available with a few fixes.

  • 2020/10/23: PyNX 2020.2 is available, notably introducing distributed computing on multiple GPU

  • 2020/09: the PyNX article: PyNX: high-performance computing toolkit for coherent X-ray imaging based on operators is out: J. Appl. Cryst. 53 (2020), 1404, also available as arXiv:2008.11511

  • 2020/02/02: version 2020.1 is out !

  • 2019/06/19: version 2019.2.6 is out. Note that 2019.2.x (x<6) versions had incorrect Ptycho CUDA scaling, preventing correct ML minimisation

  • 2018/06: all coherent imaging modules have now been converted to use an operator-based API

  • 2016/09: overhaul of the library structure - pynx.gpu has now become pynx.scattering, with pynx.scattering.gid as a sub-module.

Introduction

PyNX stands for Python tools for Nano-structures Xtallography. It can be used for:

  • Coherent X-ray imaging simulation and analysis: coherent diffraction imaging (CDI), Ptychography, Wavefront propagation, near field and far field techniques…

  • Fast scattering calculations from large number of atoms and reciprocal space positions.

PyNX is fully optimised to use Graphical Processing Units, using either CUDA or OpenCL, to provide fast calculations with 1 to 3 orders of magnitude speedup compared to standard processor calculations.

PyNX scripts

PyNX can be used simply with command-line scripts for some applications (2D/3D CDI and 2D Ptychography). These can take generic files as input, such as CXI files (http://cxidb.org), or can analyse data directly from beamlines.

PyNX as a python toolkit

PyNX can be used as a python library with the following main modules:

  1. pynx.scattering: X-ray scattering computing using graphical processing units, allowing up to 2.5x10^11 reflections/atoms/seconds (single nVidia Titan X). The sub-module``pynx.scattering.gid`` can be used for Grazing Incidence Diffraction calculations, using the Distorted Wave Born Approximation

  2. pynx.ptycho : simulation and analysis of experiments using the ptychography technique, using GPU (OpenCL). Examples are available in the pynx/Examples directory. Scripts for analysis of raw data from beamlines are also available, as well as using or producing ptychography data sets in CXI (Coherent X-ray Imaging) format.

  3. pynx.wavefront: X-ray wavefront propagation in the near, far field, or continuous (examples available at the end of wavefront.py). Also provided are sub-modules for Fresnel propagation and simulation of the illumination from a Fresnel Zone Plate, both using OpenCL for high performance computing.

  4. pynx.cdi: Coherent Diffraction Imaging reconstruction algorithms using GPU for Coherent Diffraction Imaging, in 2D or 3D, for small-angle or Bragg diffraction data. This uses either CUDA or OpenCL, but CUDA is strongly recommended for 3D data (significant speedup).

Download

PyNX is available from:

Installation

The simplest way to install PyNX (usually in an existing virtual or conda environment) is to use (in this case with both cuda and OpenCL backends):

curl -O http://ftp.esrf.fr/pub/scisoft/PyNX/pynx-latest.tar.bz2
pip install pynx-latest.tar.bz2[cuda]

For more installation options, see the detailed installation instructions (preferably using a python virtual environment)

Changelog

See the full Changelog:

Version 2022.1 (2022-02-06)

Major changes

  • simplified installation: e.g. to get both cuda and OpenCL backends, this can be done using just ‘pip install pynx.tar.bz2[cuda]’, including all dependencies.

  • Ptycho: incoherent background optimisation with a much more stable approach for AP and DM.

  • Ptycho: add near field runner script (ESRF-id16A), also enabled for simulation

  • Ptycho: much faster position optimisation (cuda). Change defaults in runner for positions optimisation (mult=5 and advise update every N cycles)

  • Ptycho: add ‘multiscan_reuse_ptycho’ option to runner scripts to chain the analysis of multiple scans (e.g. for ptycho-tomo) by re-using the previous object and probe, avoiding extra initialisations, and allowing to use a shorter algorithm chain after the first scan

  • Ptycho: the FFT scaling has been unified between CUDA, OpenCL and CPU operators which are now coherent. The amplitudes of the object and probe are each increased (compared to pre-12/2021 versions) by a factor, the product of which is equal the Fourier window size.

  • CDI: improved PSF calculation, with several options for the partial coherence kernel, including periodic update of PSF managed from the algorithm chain, loading previous PSF, etc.. The initial PSF model must be specified with the CDI runner, e.g. using psf=pseudo-voigt,0.5,0.05,20 (see doc)

  • CDI: optimisations for large data sizes, with faster calculations and less memory used. Now 960**3 datasets require ~28 GB GPU memory

  • CDI: in a python script, the object must be initialised using the InitObj operator, unless an object has been supplied.

  • CDI: add a Regrid3D class and a pynx-cdi-regrid.py script to transform small-angle CDI frames to a 3D dataset (Jérôme Kieffer)

  • CDI: the support update can raise exceptions if the support is too small or too large. This is handled automatically in the runner to restart while changing a little the threshold.

  • CDI runner: when using nb_run and nb_run_keep, also reject solutions which have a too large fraction of square modulus outside the support.

  • CDI analysis script: faster default options without phase ramp match, and parallel processing for loading an modes analysis. Copy original process information into the modes file.

  • Now requires Python>=3.7. Source installation must use ‘pip install’ and not ‘python setup.py install’

  • Switch default library for FFT to VkFFT (using pyvkfft), for both OpenCL and CUDA. scikit-cuda and gpyfft are still supported. This improves performance for opencl, and also for cuda with large FFT dimensions (>1k). Memory requirements are lowered for cuda when nx<=8192 and ny,nz<=4096

Mailing list, git & issue tracker

There is a mailing list for PyNX: to subscribe, send an email to pynx-subscribe@esrf.fr

To access the git repository and the issue tracker, you can create an account on http://gitlab.esrf and ask the developers for access to the PyNX project at http://gitlab.esrf.fr/favre/PyNX

Citation & Bibliography

If you use PyNX for scientific work, please consider including a citation:

  • If you use PyNX for coherent X-ray Imaging including CDI and ptychography:

  • If you use PyNX for GPU scattering calculations:

  • Cite the first PyNX article: J. Appl. Cryst. 44(2011), 635-640. A preprint version is also available on ArXiv:1010.2641

  • Give a link to the project: http://ftp.esrf.fr/pub/scisoft/PyNX/

PyNX re-uses or was inspired by features described in the following articles and open-source software packages:

  • PtyPy: 1. B. Enders and P. Thibault, “A computational framework for ptychographic reconstructions”, Proc Math Phys Eng Sci 472(2196), (2016).

  • M. Odstrčil, A. Menzel, and M. Guizar-Sicairos, “Iterative least-squares solver for generalized maximum-likelihood ptychography,” Optics Express 26(3), 3108 (2018).

  • S. Marchesini, A. Schirotzek, C. Yang, H. Wu, and F. Maia, “Augmented projections for ptychographic imaging,” Inverse Problems 29(11), 115009 (2013).

    1. Thibault and A. Menzel, “Reconstructing state mixtures from diffraction measurements,” Nature 494(7435), 68–71 (2013).

    1. Thibault and M. Guizar-Sicairos, “Maximum-likelihood refinement for coherent diffractive imaging,” New J. Phys. 14(6), 063004 (2012).

  • J. N. Clark, X. Huang, R. Harder, and I. K. Robinson, “High-resolution three-dimensional partially coherent diffraction imaging,” Nat Commun 3, 993 (2012).

  • S. Marchesini, “A unified evaluation of iterative projection algorithms for phase retrieval,” Review of Scientific Instruments 78(1), 011301 (2007).

License

The PyNX library is distributed with a CeCILL-B license (an open-source license similar to the FreeBSD one). See http://www.cecill.info/licences/Licence_CeCILL-B_V1-en.html

Note that CPU computing of the pynx.scattering module uses the sse_mathfun.h header, which is distributed under the zlib license. See http://gruntthepeon.free.fr/ssemath/

See http://ftp.esrf.fr/pub/scisoft/PyNX/README.txt for more details about the license, copyright, as well as other possible issues regarding ptychography.

Automated testing

To automatically test PyNX after installation, you can run the pynx-test.py script, which will run a series of tests and can help diagnose issues specific to GPU languages (OpenCL, CUDA), dependencies or applications (CDI, Ptycho..). Alternatively you can run pytest (if installed) from the root of the PyNX source directory.

Beginner tutorials

To begin using PyNX, you can read the following Tutorials:

Command-line scripts

Scripts Reference

Documentation of scripts included in PyNX

API Documentation

API Reference

Documentation of modules included in PyNX

Indices and tables