Python Software Engineering Training Materials
This site provides a combination of original resources and recommended links for Python users in
the ECP and broader scientific community. It is part of the IDEAS-ECP project.
Quick-start Guides
Presentations
Webinars
Tips on Python for Scientific Computing
Python resources for computational scientists
Scientific Libraries
Scientific computing packages in Python (running C extensions):
- NumPy NumPy is the fundamental package for scientific computing with Python.
- SciPy A Python-based ecosystem of open-source software for mathematics, science, and engineering. Now incorporates: Numpy, the SciPy library, Matplotlib, IPython, SymPy and Pandas.
Parallel and Distributed Programming Options:
Note: Python provides an in-built threading module. However, this is not really suitable for parallel computation due to the GIL (Global Interpreter Lock)
- mpi4py Python wrapper for MPI
- Multiprocessing module Basic multiple process parallelism through forked interpreters (with threading like interface). Be aware of issues mixing with OpenMP, MPI, or shared memory tools.
Python bindings to scientific libraries in HPC:
- petsc4py Python bindings for PETSc
- slepc4py Python bindings for SLEPc, the Scalable Library for Eigenvalue Problem Computations
- PyTrilinos A set of python wrappers for selected Trilinos packages
I/O:
- h5py The h5py package is a Pythonic interface to the HDF5 binary data format.
Python on Accelerators
- Numba Numba is an open source JIT compiler that translates a subset of Python and NumPy code into fast machine code (Mark hotspot functions with decorators and they are compiled.). Numba supports Intel and AMD x86, POWER8/9, and ARM CPUs, NVIDIA and AMD GPUs. Relies on decorators to identify code sections to accelerate. Requires LLVM. Works with the standard CPython. Accelerating Python with the Numba (Video)
- PyCUDA
PyCUDA lets you access Nvidia’s CUDA parallel computation API from Python.
- PyOpenCL
PyOpenCL lets you access the OpenCL parallel computation API from Python
- Parsl Use Parsl with Jupyter notebooks to scale interactive analyses from laptops to supercomputers. video
- Balsam Workflow system for managing large campaigns of interdependent jobs (unlimited queue depth). Balsam manages a database of jobs, with specified dependences. Jobs can be added to the database from anywhere on the system, for dynamic workflows.
- libEnsemble Library for dynamic ensembles using generator and simulator functions (e.g. using numerical optimization).
Conferences
Find more software engineering materials for computational scientists at the Better Scientific Software website.
Feedback
Any feedback, corrections and suggested additions are welcome: shudson@anl.gov