avaje / avaje-config

Application configuration / properties loading for JVM applications

Avaje Config

This library loads properties files that can be used to configure an application including "testing" and "local development" and dynamic configuration (changes to configuration properties at runtime).

<dependency>
  <groupId>io.avaje</groupId>
  <artifactId>avaje-config</artifactId>
  <version>${avaje.config.version}</version>
</dependency>

Typical use

• Put application.yaml into src/main/resources for properties that have reasonable defaults
• Put application-test.yaml into src/test/resources for properties used when running tests
• Specify external properties via command line arguments. These effectively override application.yaml properties.

Config use

Getting property values

// get a String property
String value = Config.get("myapp.foo");

// with a default value
String value = Config.get("myapp.foo", "withDefaultValue");

// also int, long and boolean with and without default values
int intVal = Config.getInt("bar");
long longVal = Config.getLong("bar");
boolean booleanVal = Config.getBool("bar");

Register callback on property change.

Config.onChange("myapp.foo", newValue -> {
  // do something ...
});

Config.onChangeInt("myapp.foo", newIntValue -> {
  // do something ...
});

Config.onChangeLong("myapp.foo", newLongValue -> {
  // do something ...
});

Config.onChangeBool("myapp.foo", newBooleanValue -> {
  // do something ...
});

Loading properties

Config loads properties from expected locations as well as via command line arguments. Below is the how it looks for configuration properties.

• loads from main resources (if they exist)

  • application.yaml
  • application.properties

• loads files from the current working directory (if they exist)

  • application.yaml
  • application.properties

• loads via system property props.file or environment variable PROPS_FILE (if defined)

• loads via system property avaje.profiles or environment variable AVAJE_PROFILES (if defined).

Setting the config.profiles or environment variable CONFIG_PROFILES will cause avaje config to load the property files in the form application-${profile}.properties (will also work for yml/yaml files).

For example, if you set the config.profiles to dev,docker it will attempt to load application-dev.properties and application-docker.properties.

• loads via load.properties property.

We can define a load.properties property which has name of property file in resource folder, or path locations for other properties/yaml files to load.

load.properties is pretty versatile and can even be chained. For example, in your main application properties, you can have load.properties=application-${profile:local}.properties to load based on another property, and in the loaded properties you can add load.properties there to load more properties, and so on.

Example application.properties:

common.property=value
load.properties=application-${profile:local}.properties,path/to/prop/application-extra2.properties

• loads test resources (if they exist, nb: Test resources are only visible when running tests)
  • application-test.properties
  • application-test.yaml

If no test resources were loaded then it additionally loads from "local dev" and command line:

• loads from "local dev".

We can specify an app.name property and then put a properties/yaml file at: ${user.home}/.localdev/{appName}.yaml We do this to set/override properties when we want to run the application locally (aka main method)

• loads from command line arguments

Command line arguments starting with -P can specify properties/yaml files to load

When properties are loaded they are merged/overlayed.

config.load.systemProperties

If we set config.load.systemProperties to true then all the properties that have been loaded are then set into system properties.

avaje-config's Issues

Remove support for jndi

ENH: Add enabled() as a synonym for getBool ... for better reading when used as feature toggle

if (Config.enabled("feature.cleanup")) {
  ...
}

Which is just the same as:

if (Config.getBool("feature.cleanup")) {
  ...
}

Load configuration from a specific file - is it possible?

I didn't find a way to crate a Configuration using a specific file via constructor or a specific method.
Is there a way to load an yaml file containing configuration that is stored somewhere on non default location? I saw that I can pass file name using pros.file system property or env variable, but I'd like something like Config.from(configFileName) ?

My use case is I have a single properties file that is application configuration but also ebean configuration. I'd like to use yaml and achieve the same.

Improve logging on startup - explicit "resource:" prefix for loaded resources

Add support for reading POD_NAME, POD_NAMESPACE, POD_VERSION, POD_ID

This is added as a form of convention over configuration.

The convention adopted is that in Kubernetes environment variables can be exposed:

• POD_NAME
• POD_NAMESPACE
• POD_VERSION
• POD_IP

We use these to set system properties - the most useful are appName (derived from POD_NAME) and appEnvironment (copied from POD_NAMESPACE).

---

If there is a POD_NAME env var then we can use that to set appName and appInstanceId.
If there is a POD_NAMESPACE env var then we set that as appEnvironment
If POD_IP then we set appIp
If POD_VERSION then set appVersion

ENH: Add Config.getOptional()

Add file change detection using file length

In some container file systems we can not use last modified timestamp for detection of config file changes. This change additionally uses the file length, which will only be reliable if the actual file length changes of course.

Note that in the CI build we observe this behavior.

Use "app.name" rather than "appName" for finding localdev configuration files

Use UncheckedIOException rather than IllegalStateException for consistency

In io.avaje.config.Config we have

public class Config {

  private static final Configuration data = CoreConfiguration.initialise();
...
}

Any static method called on Config will cause initialization of data. Luckily and I assume this is why @rbygrave wrote it that way is that every static method in Config happens to access data.

However another developer could easily make the mistake of adding a static method that does not need data. Perhaps a utility method used by the builder or whatever like concatPath or somethign.

Thus I recommend the canonical holder pattern of a private static inner class:

public class Config {

  private static class Holder {
    static final Configuration data = CoreConfiguration.initialise();
  }
...
}

All the static config methods would now do Holder.data for data.

Multi-Profile support

Currently, to have profiles we do.

load.properties=application-${profile:local}.properties

and set the command line args.env variables.

This is pretty good, but doesn't support multiple profiles.

I propose we add native support for profiles such that if profile=dev,docker the files application-dev.properties and application-docker.properties will be loaded

Use AppLog.getLogger() rather than System.getLogger() to allow customization

This allows for 1 extra level of indirection to control the System.Logger implementation when we can't configure it via System.LoggerFinder.

Log error when file watching but no files registered

Repeated calls to Config.get() with no value, expect to return passed default value

When there is no value for myapp.doesNotExist3 then expect:

  @Test
  public void get_default_repeated_expect_returnDefaultValue() {
    assertThat(Config.get("myapp.doesNotExist3", null)).isNull();
    assertThat(Config.get("myapp.doesNotExist3", "other")).isEqualTo("other");
    assertThat(Config.get("myapp.doesNotExist3", "foo")).isEqualTo("foo");
    assertThat(Config.get("myapp.doesNotExist3", "junk")).isEqualTo("junk");
  }

Change so that properties via command line arguments is ALWAYS read (it wasn't when test resources detected)

When test resources where detected (application-test.yaml, application-test.properties) then the command line arguments were not being read / checked.

This prevents the case where we run something in src/test (usually a test) and provide properties configuration via command line arguments. This is a pretty rare scenario, in fact I don't think anyone has hit this scenario yet but I think the existing behaviour is confusing.

Changing to: Always going to read command line arg properties.

ENH: Add Configuration.evalModify() ... to perform evaluation of expressions that modify properties in place

  /**
   * Run eval of the given properties modifying the values if changed.
   */
  void evalModify(Properties properties);

Example use:

  @Test
  void evalModify() {
    final Properties properties = basicProperties();
    properties.setProperty("someA", "before-${foo.bar}-after");
    properties.setProperty("yeahNah", "before-${no-eval-for-this}-after");

    String beforeYeahNahValue = properties.getProperty("yeahNah");

    final CoreConfiguration config = new CoreConfiguration(new Properties());
    config.evalModify(properties);

    String someAValue = properties.getProperty("someA");
    assertThat(someAValue).isEqualTo("before-42-after");

    String afterYeahNahValue = properties.getProperty("yeahNah");
    assertThat(beforeYeahNahValue).isSameAs(afterYeahNahValue);
  }

Need to filter non property command line args - IllegalArgumentException: Expecting only yaml or properties file but got [ort]

Caused by: java.lang.IllegalArgumentException: Expecting only yaml or properties file but got [ort]
at io.avaje.config.load.Loader.loadFileWithExtensionCheck(Loader.java:230)
at io.avaje.config.load.Loader.loadViaPaths(Loader.java:192)
at io.avaje.config.load.Loader.loadViaCommandLine(Loader.java:140)
at io.avaje.config.load.Loader.loadViaCommandLineArgs(Loader.java:126)
at io.avaje.config.load.Loader.loadLocalFiles(Loader.java:119)
at io.avaje.config.load.Loader.load(Loader.java:83)
at io.avaje.config.CoreConfiguration.load(CoreConfiguration.java:25)
at io.avaje.config.Config.(Config.java:18)

This is because running a test in Eclipse runs the following command...

[org.eclipse.jdt.internal.junit.runner.RemoteTestRunner, -version, 3, -port, 53186, -testLoaderClass, org.eclipse.jdt.internal.junit4.runner.JUnit4TestLoader, -loaderpluginname, org.eclipse.jdt.junit4.runtime, -test, debrepo.teamcity.ebean.server.migration.MainDbMigrationTest:testGenerateMigrationFiles]

Extracting property source.

While working on Rainbow Gum's generic agnostic property mapping validation and error message I noticed there is no way to get a CoreEntry (or analog).

That is I need the CoreEntry to get the source so that I can build better error messages.

The only way now to get the source information (e.g. like line number) is to use getAs which runs your conversion code as a Function.

Unfortunately I ended up designing Rainbow Gum where that is not really desirable as it creates very confusing call stacks.

What it instead would be easier to some wrapper object like the concrete CoreEntry so that you can do something like:

Optional<PropertyEntry> getOptionalEntry(String key). (and the non optional equivalent if you like). You can bike shed the name PropertyEntry but hopefully you get the idea.

In lightbend aka typesafe config the above analog is called ConfigValue: https://lightbend.github.io/config/latest/api/com/typesafe/config/ConfigValue.html

Maven Plugin Feature (and some other features).

Hey Rob,

Massive fan and like all your libraries.

I have been making a code generator (aka annotation processors) version of Spring's Petclinic to highlight various annotation processor libraries such as yours.

I wanted to use our own in-house config framework but I don't have the time to make it open source and accidentally stumbled on yours which has a similar initialization and design as mine. Mine has a general URI SPI mechanism, more filtering options, and the Config part is separated from the Key Value loading. There is also an annotation processor to map config to objects. It is probably massively over designed :).

Anyway here are some missing features I miss.

Feel free to let me know if anything is out of scope or not worthwhile or you want separate bugs filed for each.

Maven Plugin

One of the major features I'm missing is our config framework has a maven plugin very
much analogous to the codehaus maven properties plugin.

<plugin>
 <groupId>com.snaphop</groupId>
 <artifactId>snaphop-config-maven-plugin</artifactId>
 <executions>
   <execution>
     <phase>initialize</phase>
     <goals>
       <goal>read-config</goal>
     </goals>
     <configuration>
       <urls>
         <url>file:///${project.basedir}/src/main/resources/db/database.properties</url>
       </urls>
       <!-- finds all properties with prefix and removes it -->
       <propertiesKeyPrefix>database.</propertiesKeyPrefix>
       <!-- prefix the found properties -->
       <pomKeyPrefix>my_project.database.</pomKeyPrefix>
     </configuration>
   </execution>
 </executions>
</plugin>
<!-- now use the properties for other plugins like jOOQ or Flyway -->

I think I can trivially port our maven plugin and we could add it as a module to avaje-config (well not yet as avaje-config is not multimodule but that would be easy to fix). If you are interested I could do a PR.

More configurable initialization

On the one hand I mostly like the current opinionated behavior for simplicity on the other hand it is not exactly the canonical behavior I want. For example I prefer ~/.config/application instead of ~/.localdev. Besides the config source SPI I think a worthwhile add on would be an SPI to provide your own initialization.

Annotation processor for config objects

Another thing our config framework has is a general purpose flat mapping of key values to objects.

@ConfigBean
public interface MyClientConfig {
  public String username();
  public int port();
}

What happens is an implementation is generated that essentially will map method calls like username to Map<String,String>#get or Config#getString. I take @Nullable into account.

Anyway this is again I think something I could port to use your library.

Able to load env like files AND generate them

One of the annoying things is not everything can read java like config. Particularly shell scripts.

One of the things we have is something that will generate shell environment config from your property files config. That is initialize the framework and export out an ".env" file. This requires deciding how you want to name mangle or convert property names to environment variable names.

Side note

As a tangent I will say on the dynamic properties front which we have as well we never use anymore because reload whole app is fast these days. I wonder do you use that feature?

Add Automatic-Module-Name into MANIFEST for Java modules use

Reset last modified properly for config watching

Support TOML?

I like how it looks.

ENH: Add support for returning a Set of String, Integer and Long values

ENH: Add ConfigurationSource loaded by ServiceLoader

This allows us to plugin custom sources for configuration (like rest endpoint, redis, vaults etc).

Event System does not coalesce changes properly

I decided to file this as a feature/bug instead of continuing in the long SPI thread.

In my experience you rarely need to "react" to a single property event (property being name value pair).

Let us say I have some sort of external system with credentials and location information. Let us use a datasource ignoring the complexity reloading of database pool as an example.

I have my properties like "database.user" and "database.password" and many more. I stress many properties.

Now I load all of these guys up into some object in one event change.

How do I do the above without reacting to every single property change?

I think the event system needs to be more batching or transactional.

What I recommend is removing all the setProperties functions and potentially the onChangeXXX or deprecating them.

Here is an abbreviated version of our in house one with some name changes:

    void onEvent(Consumer<? super Event> consumer);

    // captures a snapshot of the current config to prepare for edit
    EventBuilder eventBuilder();

    void publish(Event event);

    public interface EventBuilder {

        // This snapshot is mutable and copy of the current config
        Map<String, ConfigEntry> snapshot();

        EventBuilder description(String description);

        // whether or not to take the changes of the snapshot
        EventBuilder update(boolean update);

        Event build();

    }

So instead of calling setProperty("foo", "bar") you do something like this:

eb = config.eventBuilder();
eb.snapshot().put("foo", "bar");
eb.snapshot().put("another", "one");
config.publish(eb.build());

I have simplified and removed some ergonomics as well as combined the "change request" with the downstream event but hopefully you get the idea.

Now downstream you could add information about which properties have changed but I have found that just re pulling the config you need good enough.

config.onEvent( e -> { if (e.update) service.reload(config); });

In the old system I would get an event for each property change potentially causing lots of problems particularly if the most common case is some file reload of lots of properties.

BTW to do an event system correctly otherwise bizarre shit happens is to use some sort of concurrent queue (blocking or not depending on implementation). While concurrent hashmap saves you from concurrent modifications it doesn't mean it guarantees consistency.

Anyway if you are interested I can do a PR on how I would implement it.

Improve internals for caching default values and missing values

Remove the warning log messages when yml file extension is used rather than yaml

Fix for default value containing ":" colon characters

    assertEquals("jdbc:postgresql://localhost:7432/myapp", eval("${db.url:jdbc:postgresql://localhost:7432/myapp}"));

Add a simple yaml parser that we can use make snakeyaml an optional dependency

Rename getList() getSet() methods to list() set() with deprecation

Change default file watch period to 10 seconds from 60 seconds

load.properties parameter should load from jar resources

I want to be able to load env specific props based on profile. Say I have an application properties file like this

some.common.prop=true
load.properties=src/main/resources/application-${profile:local}.properties

In the current state, files are only supported, so when I upload my jar to my server it doesn't pull the props because there is no such folder.

I'd like something like this to work as well as standard path based config

some.common.prop=true
load.properties=application-${profile:local}.properties

Bump to Java 11, use System.Logger instead of slf4j-api

Log warning when props.file is not found and loaded

Inconsistency with builder and initial loader. Default resource loader does not use the System class loader?

InitialLoader / InitialLoaderContext does several things different than what the Configuration.builder does that are very hard to replicate with the builder.

For example let us say I want to replicate most of the default initialization:
One them is this:

Configuration.builder()
.load("application.properties")
.load("application-test.properties")
.build();

There are several problems with the above. I can't specify a maybeLoad and or fallbacks. I also had the surprise that the default resourceLoader does not do:

ClassLoader.getSystemResourceAsStream which is done in io.avaje.config.InitialLoadContext:

  private InputStream resourceStream(String resourcePath) {
    InputStream is = resourceLoader.getResourceAsStream(resourcePath);
    if (is == null) {
      // search the module path for top level resource
      is = ClassLoader.getSystemResourceAsStream(resourcePath);
    }
    return is;
  }

It seems there is a lot of duplication between the builder and InitialLoadContext.

The above I would expect:

  private InputStream resourceStream(String resourcePath) {
    return resourceLoader.getResourceAsStream(resourcePath);
  }

And the default resourceLoader to handle the ClassLoader.getSystemResourceAsStream(resourcePath) fallback.

Refactor logging to use single io.avaje.config logger name

Potential NPE in CoreConfigurationBuilder

So I know Avaje does not use the newer TYPE_USE aka JSpecify-like annotations yet but you still promise to respect API and thus API has to be annotated @Nullable.

  @Override
  public Configuration.Builder load(String resource) {
    final var configParser = parser(resource);
    try {
      try (var inputStream = resourceLoader.getResourceAsStream(resource)) {
        putAll(configParser.load(inputStream)); //  <-- this is bad inputStream can be null.
        return this;
      }
    } catch (IOException e) {
      throw new UncheckedIOException(e);
    }
  }

public interface ConfigParser {
  /**
   * Parse content into key value pairs.
   *
   * @param is configuration contents
   * @return Key-Value pairs of all the configs
   */
  Map<String, String> load(InputStream is); // <-- is implicitly NonNull
}

public interface ResourceLoader {

  /**
   * Return the InputStream for the given resource or null if it can not be found.
   */
  @Nullable
  InputStream getResourceAsStream(String resourcePath);
}

The above is why I don't use JSR 305 and something stronger like Checkerframework.

I would consider seriously switching to it in the long run given I could find something like this just glancing. (I'm not trying to be rude it just is a reality).

EDIT funny I'm not even sure if it will fail at the try eagerly but still its a NPE that should not happen.

EDIT it does not fail at the try as obviously you can have null which for some reason I was not expecting but makes sense.

Change from System.Logger to use events and control how those events are logged.

What our config framework provides is a simple event system. When logging is done you can drain the queue of events usually replaying them to logging our System.err.

The reason for doing this is because sometimes we want the configuration to load completely BEFORE any logging is used. This allows the configuration to put things into System Properties which is used to configure the actual logging.

Feature request: Support configuration injection

Hi,

Could you please add something like micronaut's @Value / @Property (see micronaut's document)?

loading from .localdev supports .yaml but not .yml (Add support for .yml with .localdev configuration)

SPI for initialization

For a variety of reasons I often need to do my own initialization.

Avaje-config uses a plain static single and not the static service locator pattern like logging facades and their frameworks do.

This is problematic if you want to guarantee something always happens before or after avaje-config has loaded.

Where this usually rears its head is unit testing. It can be difficult and error prone to control the initialization of Service Locators. It's also a problem if you want to plug in a different implementation for testing or perhaps a threadlocal version for some sort of context config.

The simple solution is to make an interface like ConfigProvider or ConfigurationProvider (I'm still confused what the difference is between the two) that gets loaded with the ServiceLoader and if there are no implementations then a default one is loaded.

Consequently I recommend making an AbstractConfiguration that makes it easy for others to make their own custom Configuration implementations (again not sure if Config or Configuration is the better choice here).

ENH: Add support for returning a List of String, Integer and Long values

Add getDuration() - helper methods returning Duration

StringIndexOutOfBoundsException with multiple expressions

Failing test case

statusPageUrl=https://${eureka.instance.hostname}:${server.port}/status

java.lang.StringIndexOutOfBoundsException: begin 17, end 15, length 51

	at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3720)
	at java.base/java.lang.String.substring(String.java:1909)
	at io.avaje.config.load.CoreExpressionEval$EvalBuffer.parseForDefault(CoreExpressionEval.java:164)
	at io.avaje.config.load.CoreExpressionEval$EvalBuffer.evalNext(CoreExpressionEval.java:209)
	at io.avaje.config.load.CoreExpressionEval$EvalBuffer.process(CoreExpressionEval.java:214)
	at io.avaje.config.load.CoreExpressionEval.eval(CoreExpressionEval.java:102)
	at io.avaje.config.load.CoreExpressionEval.eval(CoreExpressionEval.java:62)
	at io.avaje.config.load.LoadContext.evalAll(LoadContext.java:129)
	at io.avaje.config.load.Loader.eval(Loader.java:238)
	at io.avaje.config.load.LoaderTest.loadProperties(LoaderTest.java:72)

Interpolation doesn't work in the same resource file.

stuff like this doesn't work

database.schema=foobar
database.host=localhost
database.portPrefix=${global.portPrefix}
database.port=${global.portPrefix}5432
database.url=jdbc\:postgresql\://${database.host}\:${database.port}/${database.schema}
database.username=${database.schema}
database.password=${database.schema} 

Funnily enough, it works if you do it programmatically via setProperty. I guess because CoreMap uses a HashMap iterating over it doesn't guarantee that the keys are read and interpolated in the correct order

Document or fix hard dependency on avaje-inject when using module-path

Hi,

when trying to evaluate avaje i could not help but notice, that there is a hard dependency on avaje-inject, albeit the require static statement in the modules-info.

A MRE can be found here:
https://github.com/NoonRightsWarriorBehindHovering/mre-avaje-config-1

The issue is expected as far as i can see, given that the official documentation says this:
A module M declares that it 'uses p.S' or 'provides p.S with ...' but package p is neither in module M nor exported to M by any module that M "reads".
Source: https://docs.oracle.com/javase/9/docs/api/java/lang/module/package-summary.html

As such without avaje-inject being read by any module explicitely (requires static does not "read", because it it optional), the above fails.

A simple workaround would be to require it in the consuming application.
This might not be feasible, when inject is not being used.
Alternatively the provides statement could (naively) be moved somewhere else (e. g. a new jar or avaje-inject).

In any case the intended outcome should be documented.

Sorry for opening two bugs already in quick succession. :)

ENH: With config.load.systemProperties: true load all properties into System properties

Add JPMS Java module support via Multi-Release jar

Refer 599836a

Add getURI() - helper methods returning URI

Bump snakeyaml dependency to 1.26

Make a Parser interface

We did dabble with this when adam asked for specialized yml parsing, but I think this is a good idea to allow for more config file types.