lundybernard / batconf

Application configuration tool from the BAT project

BatConf

Configuration Management for Python projects, modules, applications, and microservices.

Compose structured hierarchical configurations from multiple sources. Enable your code to adapt seemlessly to the current context. Allow users in different contexts to use the config source that works best for them.

• Hierarchical priority: CLI > Environment > config file > module defaults
• Provides builtin support for common config sources:
  • CLI args
  • Environment Variables
  • Config File (yaml)
  • Config classes with default values
• Easily extendable, add new sources to serve your needs.
• Set reasonable defaults, and override them as needed.
• Designed for 12-factor applications (config via Environment Variables)

Users can create their own config sources by creating classes that satisfy batconf.source.SourceInterfaceProto (or subclass batconf.source.SourceInterface)

The config lookup order is determined by the SourceList instance, which can be adjusted to suit your needs.

Example Configuration

• REF: template project example
• tests/example/ contains an example project and tests with documentation.

Most projects can copy this example with minimal modification.

from bat import GlobalConfig

from batconf.manager import Configuration, ConfigProtocol

from batconf.source import SourceList
from batconf.sources.args import CliArgsConfig, Namespace
from batconf.sources.env import EnvConfig
from batconf.sources.file import FileConfig
from batconf.sources.dataclass import DataclassConfig


def get_config(
    # Known issue: https://github.com/python/mypy/issues/4536
    config_class: ConfigProtocol = GlobalConfig,  # type: ignore
    cli_args: Namespace = None,
    config_file: FileConfig = None,
    config_file_name: str = None,
    config_env: str = None,
) -> Configuration:

    # Build a prioritized config source list
    config_sources = [
        CliArgsConfig(cli_args) if cli_args else None,
        EnvConfig(),
        config_file if config_file else FileConfig(
            config_file_name, config_env=config_env
        ),
        DataclassConfig(config_class),
    ]

    source_list = SourceList(config_sources)

    return Configuration(source_list, config_class)

GlobalConfig and Config classes

the GlobalConfig class is a python dataclass, used for namespacing, and providing a structured configuration tree. Its attributes should be other Config dataclasses for sub-modules.

from dataclasses import dataclass
from .example import Config


@dataclass
class GlobalConfig:
    # example module with configuration dataclass
    example: Config

Install Instructions

pip install .

Install with Poetry

install poetry >= 1.1.13

poetry install

Manual install

install the dev dependencies listed in pyproject.toml

Dev Guide

Install dev dependencies (pytest, mypy, etc)

pip install -e .[dev]

macos/zsh:

pip install -e ".[dev]"

Design Principles

• Non-Intrusive Integration: BatConf can be seamlessly incorporated into existing projects with minimal code modifications.
  • imports from batconf can be isolated to a single source file
  • Config classes utilize stdlib dataclasses
• Portability and Modularity: Modules (sub-modules or entire projects) that use batconf configuration should be easy to compose and refactor.
  • modules can be easily plugged in to other modules.
  • modules can be easily factored out (into new projects).

Security contact information

To report a security vulnerability, please use the Tidelift security contact. Tidelift will coordinate the fix and disclosure.

batconf's Issues

Add link to github repo in pypi page

pypi's project page should link to this repo

Add Changelog to documentation

Add a changelog to the documentation

Bump or unpin pyYAML

pyYAML needs to be unpinned, or pumped to >= 6.0.1 to fix upstream issue yaml/pyyaml#601

missing config file causes exception

When a FileConfig source is included in the Configuration source_list, but a file is not provided by the user, or found during FileConfig.init, looking up an attribute throws the following error.

  File "/venv/lib/python3.9/site-packages/batconf/manager.py", line 42, in __getattr__
    return self._get_config_opt(name, self._mod_)
  File "/venv/lib/python3.9/site-packages/batconf/manager.py", line 45, in _get_config_opt
    if value := self._config_sources.get(key, path=path):
  File "/venv/lib/python3.9/site-packages/batconf/source.py", line 17, in get
    if value := source.get(key, path):
  File "/venv/lib/python3.9/site-packages/batconf/sources/file.py", line 38, in get
    if not (conf := conf.get(k)):
AttributeError: 'str' object has no attribute 'get'

FIX: missing keys in the config file should return None, not raise an AttributeError. If the file is missing all FileConfig.get should always return None

Document release process

make release branch

update version listed in pyproject.toml

update release docs at docs/source/changelog.rst

• rebuild docs and review changelog

commit changes

bump version to next ###+dev in pyproject.toml, commit

merge into main

tag main branch with version number, stable, release on release commit

checkout release

build and publish to pypy

• pip install -e .[dev]
• poetry build
• poetry publish -r testpypi
• poetry publish

Add official support for MacOS

This project should™ work on MacOS without any issues. I have used it many frequently without any problems.

Requirements:

1. Add a MacOS test target to the github actions tests
2. Add official support indicator to the docs
3. Add official support indicator on pypi

Add credential store source

Add a new source that can search a credential store, or config database.

Motivating Example:

• An application may need to access multiple systems using similar credentials (authentication tokens), or one system using multiple users.
• Users probably want to store credentials in a secure, encrypted, service built for the purpose, and pull creds from there, we need an example of this pattern.
• we may want to store multiple configurations for a module in a database or flat-file, and use multiple variations of the same config at runtime.

Example:

the file source Already allows multiple configs at the top-level by setting its config_env parameter. We want a similar mechanism to select a sub-config at runtime.

Related to: lundybernard/project_template/issues/4

remove module/path args

Remove the path and mod arguments from the Configuration, and source .get methods. These should not be necessary, and we should be able to rely on the name argument to .get() being either a single key on that config object, or a dot-separated module path to the key.

implement __str__ for manager.Configuration object

Provide a helpful concise representation of the configuration object

It should represent the attribute tree, the value that will be returned, and source of each value
ex:

GlobalConfig
├─ submodule1
│ └─ key1: ENV -> 'value-from-env'
│ └─ key2: config.yaml -> 'value from config file'
│ └─ key3: default -> None
├─ submodule2
│ └─ key4: ARGS -> 'value from ARGS' 

Add release notes

Add a release notes section to the documentation.

• Add a release note for 0.1.8

sub-module config env_name path missing BAT_ prefix

When you have a sub_config

conf = get_config(GlobalConfig)
sub_config = conf.my_module
print(sub_config.item)

We expect the ENV path for an item to be BAT_MY_MODULE_ITEM
however, batconf.sources.env.py: EnvConfig currently looks for MY_MODULE_ITEM

We should also handle sub-modules recursively, for instance
conf.my_module.sub_module.item should lookup BAT_MY_MODULE_SUB_MODULE_ITEM
whether regardless of which config we look it up from:

conf = get_config(GlobalConfig)
print(conf.mod_config.sub_mod_config.item)
mod_config = conf.my_module
print(mod_config.sub_mod_config.item)
sub_mod_config = mod_config.sub_module
print(sub_mod_config.item)

should all lookup the same ENV variable

add install instructions to README file

add install instructions for fresh dev env to Readme

Add an AUTHORS.rst file to the project

Example from Hypothesis: https://github.com/HypothesisWorks/hypothesis/blob/master/AUTHORS.rst

Handle falsey return values from sources

batconf/batconf/source.py

Line 27 in 412297b

if value := source.get(key, path):

Currently, if a source returns a falsey value (False, None, 0, etc.) it will be ignored, and we will continue on to the next source. Falsey values can be valid configuration options, so we need to distinguish a valid None value in a config from a value Not Found None.

Test Cases:

• Yaml File sources should be able to set boolean False and None values.
• Default True values should be over-written by False values that are set in a higher priority source

Cleanup Sphinx documentation

The documentation needs to be reviewed and cleaned up.
Feel free to take on smaller tasks and open new Issues to track them.

Some things that need improvement:

• Auto-generated API documentation
• Add a detailed User Guide
• Enable make docs from the project's root directory, so we don't need to cd into docs/

Add social media preview cards to docs

The documentation published on
https://batconf.readthedocs.io/ does not provide attractive preview cards.
lets fix that.

Investigate DataclassConfig.get return values

I have some concerns about this return value.

batconf/batconf/sources/dataclass.py

Lines 27 to 42 in ffcf2fe

	def get(self, key: str, module: Ostr = None) -> Ostr:
	if module:
	path = module.split('.') + key.split('.')
	# remove the root module
	for m in self._root.split('.'):
	if m == path[0]:
	path.pop(0)
	else:
	path = key.split('.')
	# Ignore type-hinting in recursion
	conf = self._data
	for k in path:
	if (conf := conf.get(k)) is None: # type: ignore
	return conf
	return conf # type: ignore

It has the potential to return a Node (DataclassConfig) instead of a Leaf (str), which could cause unexpected behavior.

This needs further investigation.

Create and deploy project documentation

Setup documentation using Sphinx
Deploy documentation to Read the Docs

ref: getting started with sphinx

• Create a document for release notes

Fix type hints

There are several tricky type-hinting issues which MyPy reports.

• Fix the failing type-checks
• Add MyPy check to automated testing

Add path option to Configuration class

The Configuration class currently relies on the config_class __module__ attribute for its namespacing.
While this is a reasonable default/fallback, esp. for simple modules, It forces some undesirable constraints on the structure of our configuration.

Proposed improvement:

Add a path parameter to Configuration.
It should replace the _mod_ property.

• Ensure that, given a Config Dataclass, get_config can find its path, and retrieve values using it. This may require adding a __path attribute to the dataclass.

• Optional validation: Ensure the config_class path matches the path+name of the parent config.

implement __repr__ for manager.Configuration object

provide a helpful string representation of the Configuration object

Since repr should be used for debugging and unambiguous, it should display all the sources, and all of the attribute: value pairs that we know of from those sources. This can probably be accomplished fairly easily by printing the dict values of the configuration and sources recursively

ex:

<batconf.manager.Configuration object at 0x7f2c3e68e8f0>: {
    _config_class: '__main__.GlobalConfig'
    _config_sources: [
         <batconf.sources.dataclass.DataclassConfig at 0x7f2c3e68cfa0>: {
             '_root': '__main__',
             '_data': {
                 'key': 'value', 
                 'key2': None
             }
        },
    ]
}