Outflow configuration

The configuration file contains information that will generally not be versioned and more likely to change from one pipeline instance to another.

The basics

Outflow can handle json, toml and yaml file formats for the configuration file.

Here are a couple of example fields for the configuration file:

databases:
  default:
    dialect: postgresql
    admin: pipeadmin:adminpwd
    user: pipeuser:userpwd
    address: postgres
    database: pipeline_db

custom_field: my_value

Designating the configuration file

When you use Outflow, the default configuration file (config.json, config.yml or config.toml) is the one located at the root of the pipeline folder.

But, you can specify which file you’re using. Do this by using an environment variable, OUTFLOW_CONFIG_PATH, or using the CLI argument --config.

In both cases, the value should be a filepath export OUTFLOW_SETTINGS_MODULE=/path/to/my/config.yaml or --config /path/to/my/config.yaml.

Default configuration

The default Outflow configuration only includes a default logging field with a basic configuration. This configuration lives in the file outflow/core/logging/config.json.

Here’s the algorithm Outflow uses in compiling the configuration:

  • Create an empty configuration and fill the logging field with the basic logging configuration.

  • Load the configuration file from the specified filepath, overriding the default fields as necessary.

Note that if the user specify a logging field in its own configuration file, the whole default logging configuration will be overwritten.

Using the configuration in the code

In your Outflow pipeline, use config by importing the object outflow.core.pipeline.config. Example:

from outflow.core.pipeline import config

print(config.logging)

Note that outflow.core.pipeline.config isn’t a module – it’s an object. So importing individual config fields is not possible:

from outflow.core.pipeline.config import logging  # This won't work.

Altering the configuration at runtime

You shouldn’t alter the configuration in your applications at runtime. For example, don’t do this in a command:

from outflow.core.pipeline import config

config.logging = {}   # Don't do this!

[comment]: <> (## Available configuration fields) [comment]: <> (For a full list of available configuration fields, see the settings reference. (#TODO))

Creating your own configuration fields

There’s nothing stopping you from creating your own configuration fields, for your own Outflow pipeline, but don’t reinvent an already-existing field.

Customize your pipeline logs

Python’s logging library provides several techniques to configure logging, ranging from a programmatic interface to configuration files. With Outflow you can configure logging using the dictConfig format.

In order to configure logging, use the logging field of the configuration file to describe the loggers, handlers, filters and formatters that you want in your logging setup, and the log levels and other properties that you want those components to have.

Here’s a small configuration example:

logging:
  version: 1
  disable_existing_loggers: false
  handlers:
    console:
      class: logging.StreamHandler
  root:
    handlers:
    - console
    level: WARNING

Include other yaml files from your config.yml

Outflow comes with pyyaml-include that allows you to include yaml files. You might have multiple configuration files with some identical sections, this allows you to write these sections only once.