executorlib.task_scheduler.file.task_scheduler.FileTaskScheduler

class executorlib.task_scheduler.file.task_scheduler.FileTaskScheduler(executor_kwargs: dict | None = None, execute_function: ~typing.Callable = <function execute_in_subprocess>, terminate_function: ~typing.Callable | None = None, pysqa_config_directory: str | None = None, backend: str | None = None, disable_dependencies: bool = False, pmi_mode: str | None = None, wait: bool = True, refresh_rate: float = 0.01, validator: ~typing.Callable = <function validate_resource_dict>)

Bases: TaskSchedulerBase

__init__(executor_kwargs: dict | None = None, execute_function: ~typing.Callable = <function execute_in_subprocess>, terminate_function: ~typing.Callable | None = None, pysqa_config_directory: str | None = None, backend: str | None = None, disable_dependencies: bool = False, pmi_mode: str | None = None, wait: bool = True, refresh_rate: float = 0.01, validator: ~typing.Callable = <function validate_resource_dict>)

Initialize the FileExecutor.

Parameters:

• executor_kwargs (dict) – A dictionary of executor arguments required by the task. With the following keys: - cores (int): number of MPI cores to be used for each function call - cwd (str/None): current working directory where the parallel python task is executed - cache_directory (str): The directory to store cache files.

• execute_function (Callable, optional) – The function to execute tasks. Defaults to execute_in_subprocess.

• terminate_function (Callable, optional) – The function to terminate the tasks.

• pysqa_config_directory (str, optional) – path to the pysqa config directory (only for pysqa based backend).

• backend (str, optional) – name of the backend used to spawn tasks.

• disable_dependencies (boolean) – Disable resolving future objects during the submission.

• pmi_mode (str) – PMI interface to use (OpenMPI v5 requires pmix) default is None

• wait (bool) – Whether to wait for the completion of all tasks before shutting down the executor.

• refresh_rate (float) – The rate at which to refresh the result. Defaults to 0.01.

Methods

__init__([executor_kwargs, ...])	Initialize the FileExecutor.
batched(iterable, n)	Batch futures from the iterable into tuples of length n.
map(fn, *iterables[, timeout, chunksize])	Returns an iterator equivalent to map(fn, iter).
shutdown([wait, cancel_futures])	Clean-up the resources associated with the Executor.
submit(fn, /, *args[, resource_dict])	Submits a callable to be executed with the given arguments.

Attributes

future_queue	Get the future queue.
info	Get the information about the executor.
max_workers	Return the configured number of parallel workers, or None if unconstrained.

batched(iterable: list[Future], n: int) → list[Future]

Batch futures from the iterable into tuples of length n. The last batch may be shorter than n.

Parameters:

• iterable (list) – list of future objects to batch based on which future objects finish first

• n (int) – badge size

Returns:

list of future objects one for each batch

Return type:

list[Future]

property future_queue: Queue | None

Get the future queue.

Returns:

The future queue.

Return type:

queue.Queue

property info: dict | None

Get the information about the executor.

Returns:

Information about the executor.

Return type:

Optional[dict]

map(fn: Callable, *iterables, timeout: float | None = None, chunksize: int = 1)

Returns an iterator equivalent to map(fn, iter).

Parameters:

• fn – A callable that will take as many arguments as there are passed iterables.

• timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

• chunksize – The size of the chunks the iterable will be broken into before being passed to a child process. This argument is only used by ProcessPoolExecutor; it is ignored by ThreadPoolExecutor.

Returns:

map(func, *iterables) but the calls may be evaluated out-of-order.

Return type:

An iterator equivalent to

Raises:

• TimeoutError – If the entire result iterator could not be generated before the given timeout.

• Exception – If fn(*args) raises for any values.

property max_workers: int | None

Return the configured number of parallel workers, or None if unconstrained.

Returns:

The max_workers value stored in process kwargs, or None.

Return type:

Optional[int]

shutdown(wait: bool = True, *, cancel_futures: bool = False)

Clean-up the resources associated with the Executor.

It is safe to call this method several times. Otherwise, no other methods can be called after this one.

Parameters:

• wait (bool) – If True then shutdown will not return until all running futures have finished executing and the resources used by the parallel_executors have been reclaimed.

• cancel_futures (bool) – If True then shutdown will cancel all pending futures. Futures that are completed or running will not be cancelled.

submit(fn: Callable, /, *args, resource_dict: dict | None = None, **kwargs) → Future

Submits a callable to be executed with the given arguments.

Schedules the callable to be executed as fn(*args, **kwargs) and returns a Future instance representing the execution of the callable.

Parameters:

• fn (callable) – function to submit for execution

• args – arguments for the submitted function

• kwargs – keyword arguments for the submitted function

• resource_dict (dict) –

  A dictionary of resources required by the task. With the following keys: - cores (int): number of MPI cores to be used for each function call - threads_per_core (int): number of OpenMP threads to be used for each function call - gpus_per_core (int): number of GPUs per worker - defaults to 0 - cwd (str/None): current working directory where the parallel python task is executed - openmpi_oversubscribe (bool): adds the –oversubscribe command line flag (OpenMPI and

  SLURM only) - default False

  • slurm_cmd_args (list): Additional command line arguments for the srun call (SLURM only)

  • error_log_file (str): Name of the error log file to use for storing exceptions raised

    by the Python functions submitted to the Executor.

Returns:

A Future representing the given call.

Return type:

Future