MapTask

There is 3 implementations of the MapTask, one for each backend. It is recommended to use the generic outflow.core.library.tasks.MapTask and let the backend choose the right one. Additional keyword arguments are ignored, so you can for example add kwargs specific to a SlurmMapTask to the MapTask, and they will be ignored if ran with the default backend.

(SlurmMapTask)=

SlurmMapTask

You can configure how your MapTask will be distributed by passing arguments to the SlurmMapTask.

with SlurmMapTask(partition="short",
                  cpus_per_task=5,
                  mem="10GO") as map_task:
    MyComputation()

See simple-slurm github for the syntax, and sbatch documentation for available arguments.

There is only one (optional) argument specific to outflow : simultaneous_tasks which specifies the value after the % sign of the slurm array. This tells slurm how many jobs of the slurm array (ie how many mapped workflows) will be executed at the same time. This can be useful if there are limitations on the number of cpus per user on your slurm cluster.

Use config.yml to specify SlurmMapTask sbatch directives

You can easily store your SlurmMapTasks configurations in the config.yml file :

my_map_config:
  partition: batch
  cpus_per_task: 2
  simultaneous_tasks: 10
from outflow.core.pipeline import config

with SlurmMapTask(**config["my_map_config"]) as map_task:
    MyComputation()