Outflow settings

An Outflow settings file is a list of uppercase variables that will be stored in the pipeline context. This document explains how settings work and which default settings are available.

The basics

A settings file is just a Python module with module-level variables.

Here is the default Outflow settings:

import os

from outflow.core.commands import RootCommand

ROOT_DIRECTORY = os.environ.get("PIPELINE_ROOT_DIRECTORY", None)

PLUGINS = [
    "outflow.management",
]

ROOT_COMMAND_CLASS = RootCommand

Because a settings file is a Python module, the following apply:

  • It doesn’t allow for Python syntax errors.

  • It can assign settings dynamically using normal Python syntax. For example:

MY_SETTING = [str(i) for i in range(30)]

It can import values from other settings files.

Designating the settings

When you use Outflow, the default settings module is the one located at the root of the pipeline folder.

But, you can specify which settings you’re using. Do this by using an environment variable, OUTFLOW_SETTINGS_MODULE, or using the CLI argument --settings.

In both cases, the value should be in Python path syntax, e.g. export OUTFLOW_SETTINGS_MODULE=my_project.my_settings or --settings my_project.my_settings. Note that the settings module should be on the Python import search path.

Default settings

An Outflow settings file doesn’t have to define any settings if it doesn’t need to. Each setting has a sensible default value. These defaults live in the module outflow/core/pipeline/default_settings.py.

Here’s the algorithm Outflow uses in compiling settings:

  • Load settings from default_settings.py.

  • Load settings from the specified settings file, overriding the default settings as necessary.

Note that a settings file should not import from default_settings, because that’s redundant.

Using settings in Python code

In your Outflow pipeline, use settings by importing the object outflow.core.pipeline.settings. Example:

from outflow.core.pipeline import settings

if len(settings.PLUGINS) > 3:
    # Do something
    pass

Note that outflow.core.pipeline.settings isn’t a module – it’s an object. So importing individual settings is not possible:

from outflow.core.pipeline.settings import PLUGINS  # This won't work.

Also note that your code should not import from either default_settings or your own settings file. outflow.core.pipeline.settings abstracts the concepts of default settings and pipeline instance specific settings; it presents a single interface. It also decouples the code that uses settings from the location of your settings.

Altering settings at runtime

You shouldn’t alter settings in your applications at runtime. For example, don’t do this in a command:

from outflow.core.pipeline import settings

settings.PLUGINS = []   # Don't do this!

The only place you should assign to settings is in a settings file.

Available settings

PLUGINS : List of strings, each of then will be imported by outflow.

TEMP_DIR : Outflow will use this directory to save temporary files, like MapWorkflows serialized input and outputs, or workflow caches. Defaults to '/tmp/outflow_tmp. IMPORTANT : if using a shared computing cluster, /tmp is probably not shared between nodes. Please set TEMP_DIR in your settings.py file to a directory that is shared between nodes.

ROOT_DIRECTORY : Directory of the pipeline. Set by manage.py to be its current directory.

MAIN_DATABASE : Name of the default database, the one outflow will use for its internal tables, and for any models without special binds.

BACKENDS : Mapping of each backend and their module path.

ROOT_COMMAND_CLASS : Class of the root command, override for special behaviour. Defaults to RootCommand that does nothing particular.

Creating your own settings

There’s nothing stopping you from creating your own settings, for your own Outflow pipeline, but follow these guidelines:

  • Setting names must be all uppercase.

  • Don’t reinvent an already-existing setting.

For settings that are sequences, Outflow itself uses lists, but this is only a convention.