Ralph Kube
Princeton Plasma Physics Laboratory

R Michael Churchill
Princeton Plasma Physics Laboratory

Jong Youl Choi
Oak Ridge National Laboratory

Ruonan Wang
Oak Ridge National Laboratory

Scott Klasky
Oak Ridge National Laboratory

CS Chang
Princeton Plasma Physics Laboratory

Minjun J. Choi
National Fusion Research Institute, Daejeon 34133, Republic of Korea

Jinseop Park
National Fusion Research Institute, Daejeon 34133, Republic of Korea

Video: https://youtu.be/rih7Hp9nPvM

Abstract

We present Delta, a Python framework that connects magnetic fusion experiments to high-performance computing (HPC) facilities in order leverage advanced data analysis for near real-time decisions. Using the ADIOS I/O framework, Delta streams measurement data with over 300 MByte/sec from a remote experimental site in Korea to Cori, a Cray XC-40 supercomputer at the National Energy Energy Research Scientific Computing Centre in California. There Delta dispatches cython data analysis kernels using an mpi4py PoolExecutor in order to perform a spectral data analysis workflow. Internally Delta uses queues and worker threads for data communication. With this approach we perform a common spectral analysis suite on imaging measurements more than 100 times faster than with a single-core implementation.

Keywords

streaming analysis, mpi4py, queue, adios, HPC

DOI

10.25080/Majora-342d178e-013

Bibtex entry

Full text PDF

Copyright © 2020 Ralph Kube et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.