executorlib

Up-scale python functions for high performance computing (HPC) with executorlib. The executorlib module provides five different executor classes, namely: * SingleNodeExecutor - for testing executorlib on your local workstation, before up-scaling to HPC. * SlurmClusterExecutor - for SLURM clusters, submitting Python functions as SLURM jobs. * FluxClusterExecutor - for flux-framework clusters, submitting Python functions as flux jobs. * SlurmJobExecutor - for distributing Python functions within a given SLRUM job. * FluxJobExecutor - for distributing Python functions within a given flux job or SLRUM job.

In addition, the executorlib includes a BaseExecutor class to validate a given executor object is based on executorlib. Finally, the get_cache_data() function allows users to cache the content of their current cache directory in one pandas.DataFrame.

Functions

get_cache_data(cache_directory)	Collect all HDF5 files in the cache directory
get_future_from_cache(cache_directory, cache_key)	Reload future from HDF5 file in cache directory with the given cache key.
terminate_task_in_cache(cache_directory, ...)	Delete a specific job stored in the cache directory from the queuing system
terminate_tasks_in_cache(cache_directory[, ...])	Delete all jobs stored in the cache directory from the queuing system

Modules

api	External application programming interface (API) following the semantic versioning this interface is promised to remain stable during minor releases and any change in the interface leads to a major version bump.
backend	Backend for executorlib, these are the executables called by executorlib to initialize the Python processes which receive the Python functions for execution.
executor	The executor module provides five different executor classes the user interacts with, namely: SingleNodeExecutor, SlurmClusterExecutor, FluxClusterExecutor, SlurmJobExecutor and FluxJobExecutor.
standalone	Submodules in the executorlib.standalone module do not depend on other modules of the executorlib package.
task_scheduler	The task schedulers are used inside executorlib to distribute Python functions for execution, the following task schedulers are available: BlockAllocationTaskScheduler, DependencyTaskScheduler, FileTaskScheduler and OneProcessTaskScheduler.