Backend¶
Your outflow pipelines can run on different backends. There are currently 3 backends in outflow:
default
parallel
slurm
Default backend¶
The default backend executes tasks sequentially in one python process. Tasks return dictionaries and these are passed to the next tasks as parameters, so object never leave the python context. You can pass objects as big as you want between the tasks.
Parallel backend¶
The parallel backend executes tasks inside a multiprocessing pool, so independent tasks in two differente branches of a workflow can be executed at the same time. It is especially useful for the MapTask.
Slurm backend¶
The slurm backend executes MapTasks by submitting a slurm array to a slurm cluster. See SlurmMapTask for details.
Specify a backend¶
There are multiple ways to define the backend that will be used:
in the command line with the argument –backend
per command inside their definition
inside the config.yml file
Priority is in this order :
command line > command definition > config.yml
The recommended way is to leave backend: default inside the config.yml (or remove the key), and define the preferred backend for a given command using :
@RootCommand.subcommand(backend="parallel")
class MyCommand(Command):
...
To better track what is happening, you can still run this command with the default backend using : python manage.py my_command --backend default. It is useful for debugging.