Your outflow pipelines can run on different backends. There are currently 3 backends in outflow:
The default backend executes tasks sequentially in one python process. Tasks return dictionaries and these are passed to the next tasks as parameters, so object never leave the python context. You can pass objects as big as you want between the tasks.
The parallel backend executes tasks inside a multiprocessing pool. It is useful for MapWorkflow.
The slurm backend executes MapWorkflows by submitting a slurm array to a slurm cluster. See SlurmMapWorkflow for details.
Specify a backend¶
There are multiple ways to define the backend that will be used:
in the command line with the argument –backend
per command inside their definition
inside the config.yml file
Priority is in this order :
command line > command definition > config.yml
The recommended way is to leave
backend: default inside the config.yml (or remove the key), and define the preferred backend for a given command using :
@RootCommand.subcommand(backend="parallel") class MyCommand(Command): ...