Override of YML lists in externalized configuration files does not work or is documented wrongly about spring-boot

[wilkinsona]

Thanks, I understand now.

The documentation that you've referenced is referring to type-safe binding of configuration properties. The behavior where a list is only populated from a single source is specific to this binding and is what you get when you use @ConfigurationProperties. When you're accessing the environment directly, as you are doing, you aren't using the type-safe binding of configuration properties so none of its behavior applies.

If you want to mimic the same behavior when accessing the environment directly, you will have to drop down a level and work with the environment's individual property sources in order. Once you find one that contains a my-prop[] property, you'd then use it exclusively to determine all of the list's values. This is what IndexedElementsBinder does:

spring-boot/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/context/properties/bind/IndexedElementsBinder.java

Lines 66 to 75 in 9411a4c

	protected final void bindIndexed(ConfigurationPropertyName name, Bindable<?> target,
	AggregateElementBinder elementBinder, ResolvableType aggregateType, ResolvableType elementType,
	IndexedCollectionSupplier result) {
	for (ConfigurationPropertySource source : getContext().getSources()) {
	bindIndexed(source, name, target, elementBinder, result, aggregateType, elementType);
	if (result.wasSupplied() && result.get() != null) {
	return;
	}
	}
	}

The return short-circuits the binding so that only the first source the contains items for the list is used.

If you're seeing the myProp field of a MyConfigurationClass instance that's been set by @ConfigurationProperties binding contain values from multiple sources then you have found a bug. If you can provide a minimal example of it occurring we can investigate it and fix it. On the other hand, if your problem only occurs when working with the environment directly, you'll have to update your code so that it behaves in a similar manner to Boot's configuration property binding support.

[wilkinsona]

I missed your most-recent comment while writing the above. Thanks for confirming that @ConfigurationProperties is working as expected. In that case, I'm going to close this one. As you're working with the environment directly and have dropped beneath Boot's configuration properties support that builds on top of the environment, you'll have to mimic some of its functionality.

[wilkinsona]

Multiple indexed property keys mapping onto the same underlying collection-based value is a feature of type-safe configuration property binding. The Environment from Spring Framework knows nothing about this feature. As far as it's concerned my-prop[0], my-prop[1], and my-prop[2] are three different and entirely unrelated properties.

I thought that the getProperty method of the Spring Environment already takes into account the precedences of property sources

It does, however it's taken into account on a property-by-property basis.

[wilkinsona]

Assuming I have understood the problem correctly, I think this should work as you want it to. We have some tests that verify this is the case, for example:

spring-boot/spring-boot-project/spring-boot/src/test/java/org/springframework/boot/context/properties/bind/CollectionBinderTests.java

Lines 156 to 172 in f0c05a2

	@Test
	void bindToCollectionWhenMultipleSourceShouldOnlyUseFirst() {
	MockConfigurationPropertySource source1 = new MockConfigurationPropertySource();
	source1.put("bar", "baz");
	this.sources.add(source1);
	MockConfigurationPropertySource source2 = new MockConfigurationPropertySource();
	source2.put("foo[0]", "1");
	source2.put("foo[1]", "2");
	this.sources.add(source2);
	MockConfigurationPropertySource source3 = new MockConfigurationPropertySource();
	source3.put("foo[0]", "7");
	source3.put("foo[1]", "8");
	source3.put("foo[2]", "9");
	this.sources.add(source3);
	List<Integer> result = this.binder.bind("foo", INTEGER_LIST).get();
	assertThat(result).containsExactly(1, 2);
	}

spring-boot/spring-boot-project/spring-boot/src/test/java/org/springframework/boot/context/properties/bind/CollectionBinderTests.java

Lines 174 to 183 in f0c05a2

	@Test
	void bindToCollectionWhenHasExistingCollectionShouldReplaceAllContents() {
	this.sources.add(new MockConfigurationPropertySource("foo[0]", "1"));
	List<Integer> existing = new LinkedList<>();
	existing.add(1000);
	existing.add(1001);
	List<Integer> result = this.binder.bind("foo", INTEGER_LIST.withExistingValue(existing)).get();
	assertThat(result).isExactlyInstanceOf(LinkedList.class);
	assertThat(result).containsExactly(1);
	}

It's not clear where the unwanted blub is coming from in your case as it's present both in the property's default value and in application.yml. I'm also not sure where "the logic I use to get the value for the list property" comes into play. To remove these uncertainties, please provide a minimal sample that reproduces the problem and that uses a different value for the third element in the list in the two possible sources (its default and application.yml).

[Walnussbaer]

I don't think it is necessary to use a different value for the third element.

But let me summarize what the problem is:

This is my application.yml

myProp
  - machine
  - articles
  - blub

This is my application-dev.yml, It has higher precedence because it is an active profile when starting the spring boot app.

myProp
 - machine
 - articles

Now when you want to query the value in custom logic, you have to use something like this:
springEnvironment.getProperty(...)

For lists, you have to query earch list item individually by using it's index (https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.external-config.yaml). Note that using springEnvironment.getProperty("my-prop") returns null when using the yml list syntax in properties files, So I must use the index syntax.

Bascially what my logic does is the following: Iterate over all available indices to get all list values. If an index returns null, that means for me that I have reached the end of the list.

So this is what happens in my code:

this.configurableSpringEnvironment.getProperty("my-prop[0]") --> returns machine --> correct 👍 
this.configurableSpringEnvironment.getProperty("my-prop[1]") --> returns articles --> correct 👍 
this.configurableSpringEnvironment.getProperty("my-prop[2]") --> returns NOT null but blub--> NOT correct 👎 

Why is the last one not correct: because application-dev.yml only contains a list with 2 entries and not 3.

Does that clarify my problem?

[Walnussbaer]

UPDATE:

It seems like that the binding for the @ConfigurationProperties class works as expected. I just checked it in the debugger.

It only contains the 2 list items, machine and articles.

But the springEnvironment.getProperty() returns false (?) values (as described in my previous post).

[Walnussbaer]

@wilkinsona Thank you very much for the detailed explanation. I thought that the getProperty method of the SpringEnvironment already takes into account the precedences of property sources. But that doesn't seem to be case in this special scenario. Wouldn't that be better/ a reason for a new issue?

Because usually the getProperty() method returns the value with the highest precedence the spring environment contains.

For a better understanding: I have built a Config Editor for our project that enables us to alter the PropertySources and thus our @ConfigurationProperties bindings during runtime.

[Walnussbaer]

All right, that makes sense.

I think the best solution for me is to use reflection on @ConfigurationProperties classes when trying to query the current value of a configuration property in a generic way (instead of using the spring environment which I use at the moment). This way I can leverage all the features of the type-safe binding of spring boot regarding @ConfigurationProperties classes.

Thanks again for the discussion!