Matthew Rocklin
Continuum Analytics

Abstract

Dask enables parallel and out-of-core computation. We couple blocked algorithms with dynamic and memory aware task scheduling to achieve a parallel and out-of-core NumPy clone. We show how this extends the effective scale of modern hardware to larger datasets and discuss how these ideas can be more broadly applied to other parallel collections.

Keywords

parallelism, NumPy, scheduling

DOI

10.25080/Majora-7b98e3ed-013

Bibtex entry

Full text PDF

Copyright © 2015 Matthew Rocklin. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.