executorlib.task_scheduler.interactive.dependency.DependencyTaskScheduler

class executorlib.task_scheduler.interactive.dependency.DependencyTaskScheduler(executor: ~executorlib.task_scheduler.base.TaskSchedulerBase, max_cores: int | None = None, refresh_rate: float = 0.01, plot_dependency_graph: bool = False, plot_dependency_graph_filename: str | None = None, export_workflow_filename: str | None = None, validator: ~typing.Callable = <function validate_resource_dict>)

Bases: TaskSchedulerBase

ExecutorWithDependencies is a class that extends ExecutorBase and provides functionality for executing tasks with dependencies.

Parameters:

• refresh_rate (float, optional) – The refresh rate for updating the executor queue. Defaults to 0.01.

• plot_dependency_graph (bool, optional) – Whether to generate and plot the dependency graph. Defaults to False.

• plot_dependency_graph_filename (str) – Name of the file to store the plotted graph in.

• export_workflow_filename (str) – Name of the file to store the exported workflow graph in.

_future_hash_dict

A dictionary mapping task hash to future object.

Type:

Dict[str, Future]

_task_hash_dict

A dictionary mapping task hash to task dictionary.

Type:

Dict[str, Dict]

_generate_dependency_graph

Whether to generate the dependency graph.

Type:

bool

_generate_dependency_graph

Name of the file to store the plotted graph in.

Type:

str

__init__(executor: ~executorlib.task_scheduler.base.TaskSchedulerBase, max_cores: int | None = None, refresh_rate: float = 0.01, plot_dependency_graph: bool = False, plot_dependency_graph_filename: str | None = None, export_workflow_filename: str | None = None, validator: ~typing.Callable = <function validate_resource_dict>) → None

Initialize the TaskSchedulerBase.

Parameters:

• max_cores (int, optional) – Maximum number of cores available to the scheduler. Tasks requesting more cores than this will be rejected. Defaults to None (unlimited).

• validator (Callable) – Function used to validate per-task resource dicts before submission. Defaults to the no-op validate_resource_dict.

Methods

__init__(executor[, max_cores, ...])	Initialize the TaskSchedulerBase.
batched(iterable, n)	Batch futures from the iterable into tuples of length n.
map(fn, *iterables[, timeout, chunksize])	Returns an iterator equivalent to map(fn, iter).
shutdown([wait, cancel_futures])	Clean-up the resources associated with the Executor.
submit(fn, *args[, resource_dict])	Submits a task to the executor.

Attributes

future_queue	Get the future queue.
info	Get the information about the executor.
max_workers	Return the configured number of parallel workers, or None if unconstrained.

batched(iterable: list[Future], n: int) → list[Future]

Batch futures from the iterable into tuples of length n. The last batch may be shorter than n.

Parameters:

• iterable (list) – list of future objects to batch based on which future objects finish first

• n (int) – batch size

Returns:

list of future objects one for each batch

Return type:

list[Future]

property future_queue: Queue | None

Get the future queue.

Returns:

The future queue.

Return type:

queue.Queue

property info: dict | None

Get the information about the executor.

Returns:

Information about the executor.

Return type:

Optional[dict]

map(fn: Callable, *iterables, timeout: float | None = None, chunksize: int = 1)

Returns an iterator equivalent to map(fn, iter).

Parameters:

• fn – A callable that will take as many arguments as there are passed iterables.

• timeout – The maximum number of seconds to wait. If None, then there is no limit on the wait time.

• chunksize – The size of the chunks the iterable will be broken into before being passed to a child process. This argument is only used by ProcessPoolExecutor; it is ignored by ThreadPoolExecutor.

Returns:

map(func, *iterables) but the calls may be evaluated out-of-order.

Return type:

An iterator equivalent to

Raises:

• TimeoutError – If the entire result iterator could not be generated before the given timeout.

• Exception – If fn(*args) raises for any values.

property max_workers: int | None

Return the configured number of parallel workers, or None if unconstrained.

Returns:

The max_workers value stored in process kwargs, or None.

Return type:

Optional[int]

shutdown(wait: bool = True, *, cancel_futures: bool = False)

Clean-up the resources associated with the Executor.

It is safe to call this method several times. Otherwise, no other methods can be called after this one.

Parameters:

• wait (bool) – If True then shutdown will not return until all running futures have finished executing and the resources used by the parallel_executors have been reclaimed.

• cancel_futures (bool) – If True then shutdown will cancel all pending futures. Futures that are completed or running will not be cancelled.

submit(fn: Callable[[...], Any], *args: Any, resource_dict: dict[str, Any] | None = None, **kwargs: Any) → Future

Submits a task to the executor.

Parameters:

• fn (Callable) – The function to be executed.

• *args – Variable length argument list.

• resource_dict (dict, optional) – A dictionary of resources required by the task. Defaults to {}.

• **kwargs – Arbitrary keyword arguments.

Returns:

A future object representing the result of the task.

Return type:

Future