There is 3 implementations of the MapTask, one for each backend. It is recommended to use the generic
outflow.core.library.tasks.MapTask and let the backend choose the right one. Additional keyword arguments are ignored, so you can for example add kwargs specific to a SlurmMapTask to the MapTask, and they will be ignored if ran with the default backend.
You can configure how your MapTask will be distributed by passing arguments to the SlurmMapTask.
with SlurmMapTask(partition="short", cpus_per_task=5, mem="10GO") as map_task: MyComputation()
There is only one (optional) argument specific to outflow :
simultaneous_tasks which specifies the value after the % sign of the slurm array. This tells slurm how many jobs of the slurm array (ie how many mapped workflows) will be executed at the same time. This can be useful if there are limitations on the number of cpus per user on your slurm cluster.
Use config.yml to specify SlurmMapTask sbatch directives¶
You can easily store your SlurmMapTasks configurations in the config.yml file :
my_map_config: partition: batch cpus_per_task: 2 simultaneous_tasks: 10
from outflow.core.pipeline import config with SlurmMapTask(**config["my_map_config"]) as map_task: MyComputation()