Your outflow pipelines can run on different backends. There are currently 3 backends in outflow:

  • default

  • parallel

  • slurm

Default backend

The default backend executes tasks sequentially in one python process. Tasks return dictionaries and these are passed to the next tasks as parameters, so object never leave the python context. You can pass objects as big as you want between the tasks.

Parallel backend

The parallel backend executes tasks inside a multiprocessing pool. It is useful for MapWorkflow.

Slurm backend

The slurm backend executes MapWorkflows by submitting a slurm array to a slurm cluster. See SlurmMapWorkflow for details.

Specify a backend

There are multiple ways to define the backend that will be used:

  • in the command line with the argument –backend

  • per command inside their definition

  • inside the config.yml file

Priority is in this order :

command line > command definition > config.yml

The recommended way is to leave backend: default inside the config.yml (or remove the key), and define the preferred backend for a given command using :

class MyCommand(Command):