Outflow configuration¶

The configuration file contains information that is likely to change from one pipeline instance to another. It is

The basics¶

Outflow can handle json, toml and yaml file formats for the configuration file.

Here are a couple of example fields for the configuration file:

databases:
  default:
    dialect: postgresql
    admin: pipeadmin:adminpwd
    user: pipeuser:userpwd
    address: postgres
    database: pipeline_db

custom_field: my_value

Designating the configuration file¶

When you use Outflow, the default configuration file (config.json, config.yml or config.toml) is the one located at the root of the pipeline folder.

But, you can specify which file you’re using. Do this by using an environment variable, OUTFLOW_CONFIG_PATH, or using the CLI argument --config.

In both cases, the value should be a filepath export OUTFLOW_SETTINGS_MODULE=/path/to/my/config.yaml or --config /path/to/my/config.yaml.

Default configuration¶

The default Outflow configuration only includes a default logging field with a basic configuration. This configuration lives in the file outflow/core/logging/config.json.

Here’s the algorithm Outflow uses in compiling the configuration:

Create an empty configuration and fill the logging field with the basic logging configuration.
Load the configuration file from the specified filepath, overriding the default fields as necessary.

Note that if the user specify a logging field in its own configuration file, the whole default logging configuration will be overwritten.

Using the configuration in the code¶

In your Outflow pipeline, use config by importing the object outflow.core.pipeline.config. Example:

from outflow.core.pipeline import config

print(config.logging)

Note that outflow.core.pipeline.config isn’t a module – it’s an object. So importing individual config fields is not possible:

from outflow.core.pipeline.config import logging  # This won't work.

Altering the configuration at runtime¶

You shouldn’t alter the configuration in your applications at runtime. For example, don’t do this in a command:

from outflow.core.pipeline import config

config.logging = {}   # Don't do this!

[comment]: <> (## Available configuration fields) [comment]: <> (For a full list of available configuration fields, see the settings reference. (#TODO))

Creating your own configuration fields¶

There’s nothing stopping you from creating your own configuration fields, for your own Outflow pipeline, but don’t reinvent an already-existing field.

Include other yaml files from your config.yml¶

Outflow comes with pyyaml-include that allows you to include yaml files. You might have multiple configuration files with some identical sections, this allows you to write these sections only once.