GithubHelp home page GithubHelp logo

carbon-config's Introduction

Carbon Configuration

Carbon configuration provides a framework for managing server configurations. With global configuration model, server will have only one configuration file for all server configurations. So that,

  • Global configuration file (deployment.yaml) includes minimal sets of configurations which need to override very often by default (e.g: server ports etc).
  • All user settable configurations must have default values and they are burnt into compile codes.
  • Any default configuration can overridden by adding the particular configuration to the global configuration file(deployment.yaml).
  • Defining all configurations in the configuration file in not required. add only if we need to override the default value.
  • If configurations are not specified in the configuration file, values defined in the bean class will apply by default.

In global configuration file (deployment.yaml),

  • Each component will have their own configurations with a unique namespace (e.g: wso2.carbon, wso2.transports.netty etc) and it will be a fragment of deployment.yaml file. Please find the sample file below.
  • At runtime, the relevant config bean will be generated by overriding the default values specified in bean class with values mentioned in deployment.yaml file.

Sample file looks like:

  # Carbon Configuration Parameters
wso2.carbon:
  id: carbon-kernel
  version: 5.2.0-SNAPSHOT
  ports:
    offset: 0
  ...

  # Netty Transport Configurations
wso2.transports.netty:
  listeners:
    - id: msf4j-http
      host: 127.0.0.1
      port: 8080
      bossThreadPoolSize: 2
      workerThreadPoolSize: 250
      parameters:
        - name: "executor.workerpool.size"
          value: 60

...

Following annotations are introduced for configuration bean classes.

  • org.wso2.carbon.config.annotations.Configuration: This is a class-level annotation, which corresponds to a configuration bean to be used by a component.
  • org.wso2.carbon.config.annotations.Element: This is a field-level annotation, which corresponds to a field of the class.
  • org.wso2.carbon.config.annotations.Ignore: This is a field-level annotation, which specifies that the field needs to be ignored when the configuration is generated.

Sample bean class with above annotations looks like:

@Configuration(namespace = "wso2.carbon", description = "Carbon Configuration Parameters")
public class CarbonConfiguration {

    @Element(description = "value to uniquely identify a server", required = true)
    private String id = "carbon-kernel";

    @Element(description = "server name")
    private String name = "WSO2 Carbon Kernel";

    @Element(description = "server version")
    private String version = "5.2.0";

    @Ignore
    private String tenant = Constants.DEFAULT_TENANT;

}

The framework provides maven plugin to read all configuration bean classes in the component and create the relevant segment of the configuration file(for documentation purposes) automatically at compile time.

The framework provides OSGI service(ConfigProvider) to read configuration from deployment.yaml file. Following apis are read configuration from the file.

  • getConfigurationObject(Class<T> configClass): Returns configuration object of the class with overriding the values of deployment.yaml. If configuration doesn't exist in deployment.yaml, returns object with default values.
  • getConfigurationMap(String namespace): Returns configuration map of the namespace, if configuration exists for the given namespace in deployment.yaml.

For more information, Please refer document link below.

carbon-config's People

Contributors

afkham avatar arunasujith avatar aymandf avatar buddhiwathsala avatar callkalpa avatar chirangaalwis avatar cnapagoda avatar daneshk avatar dinushab avatar imesh avatar jsdjayanga avatar kasunbg avatar keerthu avatar kishanthan avatar lakshmanudayakantha avatar manoj-kristhombu avatar manuri avatar mohanvive avatar nilminiwso2 avatar nipuni avatar niranjan-k avatar niveathika avatar pcnfernando avatar praveennadarajah avatar sameerajayasoma avatar shan1024 avatar susankha avatar udarakr avatar vidurananayakkara avatar wso2-jenkins-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

carbon-config's Issues

Add document to describe how to use placeholder in configs

Currently there are three placeholder can be used in deployment.yaml configs
${sys:} - read system property
${env:} - read environment property
${sec:} - read secure vault encrypted values

Add a document describing how to use above placeholders

Move Configuration Provider from Kernel

In Carbon 5.2.0-m3, configuration provider and org.wso2.carbon.extensions.configuration.maven.plugin resides in kernel and supports only OSGi model. This needs to be moved to this repo and in-addition to OSGi support, it should also provide support for non-OSGi.

deployment.yaml should be runtime specific, i.e., <CARBON HOME>/wso2/<runtime>/config

Test Dependencies are not in test scope

Following test dependencies are not in "test" scope in config component.

        <dependency>
            <groupId>org.easymock</groupId>
            <artifactId>easymock</artifactId>
        </dependency>
        <dependency>
            <groupId>org.testng</groupId>
            <artifactId>testng</artifactId>
	</dependency>

Please have a look.

Change Environment Variable definition

Description:
In order to define an environment variable for name config value under complexBean, we have to defind environment variable as follows.

WSO2.CONFIGURATION_COMPLEXTESTBEAN_NAME="default"

As OS its not support to keep (.) mark in the environment variable due to that we had convert it to( _ ) in the namespace.

current configuration as follows

WSO2_CONFIGURATION_COMPLEXTESTBEAN_NAME="default"
wso2.configuration:
  complexBean:
    name: default
    Bean:
      name: default

Server startup error in Windows

Description:
Starting up the SP server in any runtime is throwing up the following error. The error appears to be caused by the kernel not resolving the ${sys:carbon.home} property (i.e. missing \ characters):
Bad pathname"; "E:wso2sp-4.0.0-Alpha3-SNAPSHOTbin../wso2/editor/database/WSO2_CARBON_DB.mv.db"

The full error is as below:
[2017-11-24 04:08:52,874] ERROR {org.wso2.carbon.kernel.internal.startupresolver.StartupComponentManager} - RuntimeException occurred while calling onAllRequiredCapabilitiesAvailable of component carbon-datasource-service com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization: IOException: "java.io.IOException: Bad pathname"; "E:wso2sp-4.0.0-Alpha3-SNAPSHOTbin../wso2/editor/database/WSO2_CARBON_DB.mv.db" [90031-196] at com.zaxxer.hikari.pool.HikariPool.initializeConnections(HikariPool.java:581) at com.zaxxer.hikari.pool.HikariPool.(HikariPool.java:152) at com.zaxxer.hikari.HikariDataSource.(HikariDataSource.java:73) at org.wso2.carbon.datasource.rdbms.hikari.HikariRDBMSDataSource.getDataSource(HikariRDBMSDataSource.java:56) at org.wso2.carbon.datasource.rdbms.hikari.HikariDataSourceReader.createDataSource(HikariDataSourceReader.java:74) at org.wso2.carbon.datasource.core.DataSourceBuilder.buildDataSourceObject(DataSourceBuilder.java:79) at org.wso2.carbon.datasource.core.DataSourceBuilder.buildDataSourceObject(DataSourceBuilder.java:60) at org.wso2.carbon.datasource.core.DataSourceBuilder.buildCarbonDataSource(DataSourceBuilder.java:44) at org.wso2.carbon.datasource.core.DataSourceManager.initDataSources(DataSourceManager.java:153) at org.wso2.carbon.datasource.core.internal.DataSourceListenerComponent.onAllRequiredCapabilitiesAvailable(DataSourceListenerComponent.java:125) at org.wso2.carbon.kernel.internal.startupresolver.StartupComponentManager.lambda$notifySatisfiableComponents$7(StartupComponentManager.java:250) at java.util.ArrayList.forEach(ArrayList.java:1249) at org.wso2.carbon.kernel.internal.startupresolver.StartupComponentManager.notifySatisfiableComponents(StartupComponentManager.java:236) at org.wso2.carbon.kernel.internal.startupresolver.StartupOrderResolver$1.run(StartupOrderResolver.java:204) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Caused by: org.h2.jdbc.JdbcSQLException: IO Exception: "java.io.IOException: Bad pathname"; "E:wso2sp-4.0.0-Alpha3-SNAPSHOTbin../wso2/editor/database/WSO2_CARBON_DB.mv.db" [90031-196] at org.h2.message.DbException.getJdbcSQLException(DbException.java:345) at org.h2.message.DbException.get(DbException.java:168) at org.h2.message.DbException.convertIOException(DbException.java:330) at org.h2.store.fs.FilePathDisk.toRealPath(FilePathDisk.java:204) at org.h2.store.fs.FilePathDisk.toRealPath(FilePathDisk.java:34) at org.h2.store.fs.FileUtils.toRealPath(FileUtils.java:77) at org.h2.engine.ConnectionInfo.getName(ConnectionInfo.java:412) at org.h2.engine.Engine.openSession(Engine.java:49) at org.h2.engine.Engine.openSession(Engine.java:176) at org.h2.engine.Engine.createSessionAndValidate(Engine.java:154) at org.h2.engine.Engine.createSession(Engine.java:137) at org.h2.engine.Engine.createSession(Engine.java:27) at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:354) at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:116) at org.h2.jdbc.JdbcConnection.(JdbcConnection.java:100) at org.h2.Driver.connect(Driver.java:69) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95) at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101) at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:496) at com.zaxxer.hikari.pool.HikariPool.initializeConnections(HikariPool.java:565) ... 15 more Caused by: java.io.IOException: Bad pathname at java.io.WinNTFileSystem.canonicalize0(Native Method) at java.io.WinNTFileSystem.canonicalize(WinNTFileSystem.java:428) at java.io.File.getCanonicalPath(File.java:618) at org.h2.store.fs.FilePathDisk.toRealPath(FilePathDisk.java:201) ... 31 more

Suggested Labels:
Blocker

Suggested Assignees:

Affected Product Version:

OS, DB, other environment details and versions:
Windows 7, Windows 10.

Steps to reproduce:
Run the server in windows environment

Related Issues:
wso2/product-sp#163, wso2/carbon-kernel#1615

[Standalone Mode] ConfigProvider should able to work when secure vault is not configured

At the moment carbon-config has a tight dependency with carbon-secvault. It is Ok to have a tight dependency with secvault but there are use cases where we don't require secure vault but only required config provider functionality. But, it seems config provide will not work unless we provide necessary files (master-keys.yaml and secret.properties) which are related secvault to the processing. We need to fix this..

Below is the error that we could observe.

[main] ERROR org.wso2.carbon.databridge.agent.AgentHolder - Unable to complete initialization of agents.Error while loading the configuration file /home/mohan/wso2/source-code/mohan-repo/carbon-analytics-common/components/data-bridge/org.wso2.carbon.databridge.agent/src/test/resources/data.agent.config.yaml
org.wso2.carbon.databridge.agent.exception.DataEndpointAgentConfigurationException: Error while loading the configuration file /home/mohan/wso2/source-code/mohan-repo/carbon-analytics-common/components/data-bridge/org.wso2.carbon.databridge.agent/src/test/resources/data.agent.config.yaml
at org.wso2.carbon.databridge.agent.AgentHolder.loadConfiguration(AgentHolder.java:182)
at org.wso2.carbon.databridge.agent.AgentHolder.(AgentHolder.java:65)
at org.wso2.carbon.databridge.agent.AgentHolder.getInstance(AgentHolder.java:81)
at org.wso2.carbon.databridge.agent.DataPublisher.(DataPublisher.java:125)
at org.wso2.carbon.databridge.agent.test.binary.OneEndPointDPBinaryTest.testInvalidAuthenticationURLs(OneEndPointDPBinaryTest.java:189)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:659)
at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:845)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1153)
at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:108)
at org.testng.TestRunner.privateRun(TestRunner.java:771)
at org.testng.TestRunner.run(TestRunner.java:621)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:357)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:352)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:310)
at org.testng.SuiteRunner.run(SuiteRunner.java:259)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1199)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1124)
at org.testng.TestNG.run(TestNG.java:1032)
at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:281)
at org.apache.maven.surefire.testng.TestNGXmlTestSuite.execute(TestNGXmlTestSuite.java:75)
at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:121)
at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121)
Caused by: org.wso2.carbon.databridge.agent.exception.DataEndpointAgentConfigurationException: Error in when loading databridge agent configuration
at org.wso2.carbon.databridge.agent.AgentHolder.loadConfiguration(AgentHolder.java:150)
... 31 more
Caused by: org.wso2.carbon.config.ConfigurationException: Error while loading securevault service
at org.wso2.carbon.config.ConfigProviderFactory.getSecureVault(ConfigProviderFactory.java:106)
at org.wso2.carbon.config.ConfigProviderFactory.getConfigProvider(ConfigProviderFactory.java:49)
at org.wso2.carbon.databridge.agent.AgentHolder.loadConfiguration(AgentHolder.java:139)
... 31 more
Caused by: org.wso2.carbon.secvault.exception.SecureVaultException: Error initializing securevault, secure configuration does not exist
at org.wso2.carbon.secvault.SecureVaultUtils.getSecureVaultConfiguration(SecureVaultUtils.java:245)
at org.wso2.carbon.secvault.SecureVaultUtils.resolveFileToString(SecureVaultUtils.java:224)
at org.wso2.carbon.secvault.SecureVaultUtils.getSecureVaultConfig(SecureVaultUtils.java:88)
at org.wso2.carbon.secvault.SecureVaultFactory.getSecureVault(SecureVaultFactory.java:49)
at org.wso2.carbon.config.ConfigProviderFactory.getSecureVault(ConfigProviderFactory.java:103)
... 33 more

Cannot generate Configuration Object which doesn't have namespace

Description:
With the current implementation, in order to generate bean object from configuration provider we need to add namespace to the bean class. for that we need to add config provider dependency for the component. but add annotation for bean class is not possible every time. If we cannot add namespace, we cannot retrieve the object from the config provider.

So we need to provide a way to generate bean object by given namespace rather not depending on the class namespace.

Suggested Labels:
enhancement

Suggested Assignees:
daneshk

Affected Product Version:
2.1.2

Add List support in ConfigProvider service

Current ConfigProvider service only support to return configuration bean object and configuration Map. according to below example, we are expecting List instead to Map. We need to add List support to cater the requirement.

  # Data Sources Configuration
wso2.datasources:
  - name: WSO2_CARBON_DB
    description: The datasource used for registry and user manager
      # JNDI mapping of a data source
    jndiConfig:
        # JNDI name
        # THIS IS A MANDATORY FIELD
      name: jdbc/WSO2CarbonDB/test
        # JNDI Reference Flag
      useJndiReference: false

Resolve jacoco maven plugin - class already instrumented exception when building the project

Jacoco plugin throws class already instrumented exception when building the project. These exceptions should not be thrown and should be resolved

More information:

A sample exception will be as shown below:

Caused by: java.lang.IllegalStateException: Class org/wso2/carbon/config/configuration/ConfigurationUtils is already instrumented.
	at org.jacoco.agent.rt.internal_b0d6a23.core.internal.instr.InstrSupport.assertNotInstrumented(InstrSupport.java:89)
	at org.jacoco.agent.rt.internal_b0d6a23.core.internal.instr.ClassInstrumenter.visitField(ClassInstrumenter.java:55)
	at org.jacoco.agent.rt.internal_b0d6a23.asm.ClassVisitor.visitField(ClassVisitor.java:272)
	at org.jacoco.agent.rt.internal_b0d6a23.asm.ClassReader.readField(ClassReader.java:768)
	at org.jacoco.agent.rt.internal_b0d6a23.asm.ClassReader.accept(ClassReader.java:689)
	at org.jacoco.agent.rt.internal_b0d6a23.asm.ClassReader.accept(ClassReader.java:506)
	at org.jacoco.agent.rt.internal_b0d6a23.core.instr.Instrumenter.instrument(Instrumenter.java:83)
	at org.jacoco.agent.rt.internal_b0d6a23.core.instr.Instrumenter.instrument(Instrumenter.java:101)
	... 47 more

This exception, however, does not impact the build or jacoco reports in any way. The plugin is configured to proceed ignore this warning and produce jacoco reports using the following config

<execution> <id>default-restore-instrumented-classes</id> <goals> <goal>restore-instrumented-classes</goal> </goals> </execution>

The config provide plugin need to copy the config-docs/ folder into the target/classes/ folder

Moving Issue from Kernel, wso2/carbon-kernel#1315

This is regarding the the maven plugin - 'org.wso2.carbon: org.wso2.carbon.extensions.configuration.maven.plugin'

At the moment, the generated config-docs is copied directly inside target/ folder. We need to add the config-docs inside target/classes/ folder by default. The rationale is, the contents under src/main/resources already goes to target/classes/ folder. The config-docs is another resource file that should be packed within the bundle itself.

Going forward, we could also provide an optional config to provide the outputDirectory. Still, the default should be to go inside target/classes.

Incorrect description get generated when there is more than one new line characters together.

Description:
Weird doc is getting generated when there are more than one newline characters together in a description.

Suggested Labels:

Suggested Assignees:

Affected Product Version:
2.0.5
OS, DB, other environment details and versions:
JDK 1.8
Steps to reproduce:
When we have a description like below in a @element annotation,
@Element(description = "Within Andes there are content chunk handlers which convert incoming large content\n" + " chunks into max content chunk size allowed by Andes core. These handlers run in parallel\n" + " converting large content chunks to smaller chunks.\n" + "\n" + " If the protocol specific content chunk size is different from the max chunk size allowed\n" + " by Andes core and there are significant number of large messages published, then having\n" + " multiple handlers will increase performance.")

Generated doc was like this.
screen shot 2017-10-13 at 2 46 40 pm

Related Issues:

Sys value does not work if set to default value in the Bean class

If the default value in the Bean class is set using the environment, say "${sys:carbon.home}, then it does not work. For example in carbon-deployment, the repositoryLocation is obtained from the deployment.yaml. If the default value is set as "${sys:carbon.home}/deployment/" then it does not work.

Implement configuration lookup from environment variable

From the current approach, when we need to read the configurations from environment variables, we need to specifically mention in the deployment.yaml file as

gateWayEndpoint: ${env:gateWayEndpoint}

But in a containerized environment, changing configuration file is harder, because we cannot maintain a single Container Image for different deployments, and manipulate the Container runtimes via environment variables.

From the suggested approach, read priority will be,

  1. Read configurations from environment variables. environment variable should define in a certain format like

PREFIX_wso2.NAMESPACE_CONFIGKEY="value", PREFIX_wso2.NAMESPACE_CONFIGPARENT_CONFIG2="value2"

  1. Read the deployment.yaml file.
  2. Get default values from the bean class.

Project Dependencies and Project are not picked in Carbon Configuration Maven Plugin

Carbon configuration maven plugin does not pick the project dependencies and the project when building the configuration. This will cause the plugin not to build the configuration when the specified configuration class is within the project or in a specified dependency

Example:

When the plugin is specified as mentioned below

           <plugin>
                <groupId>org.wso2.carbon.config</groupId>
                <artifactId>org.wso2.carbon.config.configuration.maven.plugin</artifactId>
                <version>${carbon.configuration.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>create-doc</goal>
                        </goals>
                        <phase>compile</phase>
                    </execution>
                </executions>
                <configuration>
                    <configclasses>
                        <configclass>org.wso2.carbon.kernel.config.model.CarbonConfiguration</configclass>
                    </configclasses>
                </configuration>
            </plugin>

and dependencies as mentioned below:

    <dependencies>
        <dependency>
            <groupId>org.wso2.carbon</groupId>
            <artifactId>org.wso2.carbon.core</artifactId>
        </dependency>
    </dependencies>

Maven configuration plugin throws a "NoClassDefFoundError" stating that the org.wso2.carbon.kernel.config.model.CarbonConfiguration (specified configuration classes) definition is not found. This happens because the org.wso2.carbon.core is not in the class realm. Ideally, dependencies should be added to the class realm so that users can specify the dependencies for which they want to run the config plugin

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.