hapipal / confidence

Dynamic, declarative configurations

confidence

Dynamic, declarative configurations

Lead Maintainer: Sunny Bhanot

Installation

npm install @hapipal/confidence

Usage

See also the API Reference

Confidence is intended for use with nodejs v12+ (see v4 for lower support).

Confidence is a configuration document format, an API, and a foundation for A/B testing. The configuration format is designed to work with any existing JSON-based configuration, serving values based on object path ('/a/b/c' translates to a.b.c). In addition, Confidence defines special $-prefixed keys used to filter values for a given criteria.

Example

Below is an example configuring a hapi server using a dynamic Confidence configuration.

const Hapi = require('@hapi/hapi');
const Confidence = require('@hapipal/confidence');

const store = new Confidence.Store({
    server: {
        host: 'localhost',
        port: {
            $param: 'PORT',
            $coerce: 'number',
            $default: 3000
        },
        debug: {
            $filter: 'NODE_ENV',
            $default: {
                log: ['error'],
                request: ['error']
            },
            production: {
                request: ['implementation']
            }
        }
    }
});

const config = store.get('/', process.env);

const server = Hapi.server(config);

Extras

Confidence originated in the hapijs organization, and was adopted by hapi pal in April 2019.

Logo

confidence's Issues

Add `-a code`

Add -a code to the Makefile

Attributes containing special characters

Let's say we have a json document such as:

{
    "things": {
        "sub-thing": "space race"
    }
}

I'm getting undefined when calling conf.get('/things/sub-thing').

I've tried a handful of escaping attempts. I also quickly scanned the lib but didn't grok anything that seemed relevant to my goal. I also didn't notice any test cases with special characters.

upgrade to hoek 2.x.x

Please upgrade to hoek 2.x.x version

Clean up Tests

Clean up tests in accordance with outmoded/discuss#24

filter object

let config = { 
   database: {
        development: {
            database: 'create_trip',
            username: 'root',
            password: 'root',
            options: {
                host: 'localhost',
                dialect: 'mysql'    
            }
        },
        '$filter': 'env',
        '$default': 'development'
    }
}

const store = new Confidence.Store(config);

store.get('/database') => 'development'

how can i get entire object instead 'development'

Is it possible to add a key to an object depending on a value ?

I would like to have conditionally add the "blipp" subobject to my "plugins" object with a filter.

Have as a result when phase=LOCAL:

"plugins": {
  "blipp": {
    "showAuth": true
  },
  "other": {
  }
}

Have as a result when phase!=LOCAL:

"plugins": {
  "other": {
  }
}

replace yargs with bossy

Share keys above $filter

Is it possible to share keys? I find myself repeating keys that I would normally only write once.

example:

var config = {
  api: {
    $filter: 'env',
    production: {
      root: process.env.API_ROOT,
      log_file: './api/hapi.log',
      session_cookie_name: 'sid',
      session_secret: process.env.SESSION_SECRET
    },
    $default: {
      root: process.env.DEV_API_ROOT,
      log_file: './api/hapi.log',
      session_cookie_name: 'sid',
      session_secret: process.env.DEV_SESSION_SECRET
    }
  }
}

Here 'log_file' is repeated. It would be nice if I could put it above the $filter and allow it to share with both 'env''s.

example:

var config = {
  api: {
    log_file: './api/hapi.log',
    $filter: 'env',
    production: {
      root: process.env.API_ROOT,
      session_cookie_name: 'sid',
      session_secret: process.env.SESSION_SECRET
    },
    $default: {
      root: process.env.DEV_API_ROOT,
      session_cookie_name: 'sid',
      session_secret: process.env.DEV_SESSION_SECRET
    }
  }
}

bad control chars when using confidence command line

When doing a npm install confidence (latest version):

[email protected] node_modules/confidence
├── [email protected]
├── [email protected] ([email protected])
├── [email protected] ([email protected], [email protected])
└── [email protected] ([email protected], [email protected])

I took the example in the confidence readme.

$ cat -vt config.json
{
"key1": "abc",
"key2": {
"$filter": "system.env",
"production": 1
}
}
when running node_modules/.bin/confidence --filter.system.env=qa -c config.json > result.json
I get this:

$ cat -vt result.json
{
^I"key1": "abc"
}

Note the extra control character that wasn't in the original file. This causes issues when trying to load this into hapi later, because it is invalid json. I have a more complex setup that I use and I get the following error when its auto generated but that file has a ton of ^I chars as well:

[1] servers position 0 fails because client is required
at Object.exports.assert (../node_modules/hoek/lib/index.js:425:11)
at Object.exports.assert (../node_modules/hapi/lib/schema.js:15:10)
at Function.internals.Pack.compose (../node_modules/hapi/lib/pack.js:699:12)
at ../node_modules/hapi/lib/cli.js:112:19
at ../node_modules/hapi/lib/cli.js:94:9
at LOOP (fs.js:1358:14)
at process._tickDomainCallback (node.js:459:13)

This seems to have broken for me sometime on August 13th. It worked fine in the morning but broke sometime in the afternoon if that is helpful. I suspect a dependency since I don't think the top level version was changed.

Note: I did get around the problem by adding a | tr -d '\t' before I write the file, and that lets it start up just fine.

$coerce array

I sometime deal with environment variables storing an array of string that I want to split. I currently do that outside of confidence.

I thought of having a way to declare it like :

new Confidence.Store({
    foo : {
        bar: { $env : 'MY_ENV', $coerce : 'array', $splitToken : ',', $default : [] },
    }
})

Or would it be better to keep this out of Confidence ? As it can introduce some weird edge case like wanting to coerce to an array of number. It could also lead to more feature like parsing JSON, etc...

I can send a PR to implement the feature.

Confidence storing a javascript object

Im using confidence to store plugins and routes in my manifest.json, but because confidence doesnt allow javascript object to be held (and therefore passed as plugin parameters to a plugin, like 'hapi-sequelize' for example) I cannot store my register object for hapi-sequelize in my manifest.json.

To workaround this issue i registered the hapi-sequelize plugin directly, but I wish to know if there is a way for this to work smoothly (register hapi sequelize inside my manifest.json), or any planned support for instances inside of parameters to plugins through Confidence?

The specific problem with hapi-sequelize is that the plugin requires a sequelize instance that cant be passed in through confidence.

Or maybe I am just doing something completely wrong. I hope so.

Thank in advance

Update deps

and a little housekeeping

using in webpack with uglifyjs breaks uglification

any ideas on this? I like the filtering functionality and was wanting to use it with webpack.

Manifest TLS options never work

Hi,

I am trying to use TLS options in manifest connections but it never worked for HTTPS, please guide what is wrong in below code whereas standard server creation works fine

Allow pulling values from criteria into store

It would be nice to be able to take values in criteria and use them as parameters for store values. Here's an example demonstrating how it might look,

const store = new Confidence.Store({
    c: {
        $filter: 'size',
        big: { $param: 'maxSize' },
        small: 1,
        $default: 50
    }
});

store.get('/c', { size: 'big', maxSize: 200 }); // Evaluates to 200

If this sounds like a useful feature I would be happy to write some code for it. Let me know if you'd like more information about my use-case.

Drop support for node v10 and below

Now that we're all tested on node v14, it's time to look ahead.

As part of this work, we should also update @hapi/* dependencies that were stuck on old versions due to our support of old node versions.

New maintainer?

Project is not being maintained to hapi.js community standards. Too many open issues and PRs.

Should we remove Confidence.id.generate() and Confidence.id.criteria()?

Confidence has two undocumented (but public) methods Confidence.id.generate() and Confidence.id.criteria(). Their purpose is to generate a guid and then use that guid to generate random criteria that can be used in the context of A/B testing. For example, Confidence.id.generate() could generate a guid to assign to a user's session, and that guid seeds random criteria values to obtain a random (but stable) configuration for that user. This way the user receives a random experience based on the confidence configuration, but the experience doesn't change for them over time.

This isn't documented, pal does not use confidence in this way (in the boilerplate), and I cannot find any issues related to it in github, so I wonder if we can safely part ways with this functionality in v6.

Why Boom dependency ?

Hi,

I am using confidence with some Hapi projects but as well with other client side projects (React Native) and I was surprised to see that the project is pulling Boom as dependency which makes no sense outside a server context.

Do we really need to encapsulate confidence error with Boom ? Could we make it pluggable and have a default behaviour that would not decorate error with Boom ?

Thanks.

Regex values values become objects

The keys for test become {} instead of remaining regex value.

multiple options for keys, not just values based on filters

What if I have:

{
  "plugins": {
    "$filter": "env",
    "$base": {
      "inert": {}
    },

    "qa": {
      "blipp": {}
    },
    "prod": {
      "good": {}
    },

    "$filter": "debug",
    "prod": {
      "debug": {}
    }
    "qa": {
      "extra-debug": {}
    }
  }
}

Right now it's very easy to have values change based on particular filters, however, keys at the same level doesn't seem to have the ability to change. For instance, I may want to have extra debugging plugin loaded for qa. It would be nice to have multiple filters at the same key level.

I don't believe this is possible atm?

( yes I realize in this instance you could argue that a debug plugin should be able to take options for debug levels, but this is an example and not the actual use-case, so please bear with me )

Random ordering occurs when there is a dash '-' in element key names

We use confidence for processing package.json files, these usually have packages with dashes in the names, e.g. good-broadcast
When these files are processed by confidence, the output has element keys reordered in a seemingly random fashion.
just try the following input data:

{
    "A-1": "I should be first", 
    "B": "runner up", 
    "C": 50 
}

This randomization can occur at any level of nesting within the input structure so long as there is a dash in a key name.

This is severely hampering our ability to determine clear differences between subsequent applications of confidence. What's worse is if there is no conditional processing required, the output is still jumbled.

Null values when filtering array items

To reproduce:

1. add to test.json

{
  "plugins": [
    {
      "$filter": "env",
      "local": "here"
    },
    {
      "$filter": "env",
      "stage": "stage"
    }
  ]
}

2. confidence -c test.json --filter.env=stage

Expected result:

{
    "plugins": [
        "stage"
    ]
}

Actual result:

{
    "plugins": [
        null,
        "stage"
    ]
}

Note that running confidence -c test.json --filter.env=local works as expected, producing:

{
    "plugins": [
        "here"
    ]
}

with no null value for the second entry

Reorganize documentation for org-wide consistency

I have been reorganizing documentation over the past few months so that each module is consistent. That means adding an installation section at the top of the readme, adding an API.md, placing the notice about node support in the usage section of the readme, etc. See toys' README.md and API.md files as an example: https://github.com/hapipal/toys.

Move from yargs to bossy

We use yargs in the CLI primarily because the filter CLI argument needs to be a dynamic object. If hapijs/bossy#77 lands then I believe we will be able to remove yargs from this project and replace it with bossy. This would be great because it will make this consistent with hpal, hpal-debug, lab, etc. Given that confidence is part of the boilerplate, this also has the benefit of keeping transitive dependencies for basic hapi pal setups minimal and within the pal/hapi orgs as much as possible.

See also #31.

Issue when passing fs.readFileSync buffer

I was working on getting TLS to work with frame that uses Glue+Manifest+Confidence for configuration management. Having tried both key/cert and pfx pattern and days of debugging, I believe the issue of not having TLS config correctly passed to the underlying node createServer was caused by Confidence when converting the config file.

Although hapi's documentation stated that the tls setting should be directly passed to node using fs.readFileSync, Confidence converted these binary file buffer to a strange format that 1. couldn't be read correctly but compiled and didn't throw error or 2. caused failed to create BIO error. The tricky thing is that the buffer will get half read and I could only debug the tls handshake failure using openssl to pinpoint the issue.

The current workaround of encoding the binary buffer to utf8 would only work with key/cert pattern and not with pfx since the converted string would be too large and node will throw a header too long exception.

In addition, this issue was further covered by node's complete conversion to TLSv1+ and wouldn't throw an error when using a SSLv2-3 key/cert. But this only pertains to my own debugging experience.

The apparent solution would be to create a special case for server.connection.tls and pass it without pre-processing. The root cause however I think would be handling raw buffer in general.

Please see this issue on glue to get a better sense of problem. I originally believed that the issue was caused by glue, but after just doing bare-bones testing, which is use only hapi with Confidence and glue separately, I've come to the conclusion that Confidence caused this issue. You can replicate this issue by simply writing a https server with Confidence and hapi, and read in tls key/cert directly without encoding to utf8.

Thanks!

Either allow matching "$undefined" or any non-falsy criteria value

Currently there's no way to only have a $default value if the filtered criteria is undefined.

e.g.

{
  "$filter": "env.canBeUndefined",
  "$default": "env:canBeUndefined",
  "$undefined": "defaultValue"
}

(env: is shortstop syntax to get an env var)

We could also just have $truthy,$falsy,$null, etc but that's probably taking it too far.

An alternative to this is just manipulating criteria before passing it in but we just want to use process.env as it is.

I think #67 is about pretty much the same thing but in a different way.

Replace ALCE file format usage in CLI

The CLI allows users to reference ALCE-formatted files: essentially a more flexible form of JSON that allows comments. There are a few downsides to using this format: 1. it is not a widely-known or -used format 2. as such the alce library hasn't been touched in 5 years (e.g. to run tests on newer versions of node) and 3. it requires some pretty heavy dependencies such as esprima.

I see three paths forward:

• Move to the json5 format which is actively supported and has the same purpose as ALCE.
• Only support JSON files.
• Instead require() the config file to support JSON or a module export. In this case we could support export of a confidence store in addition to a plain object.

My recommendation would be the last option, since it's both flexible and simple. It would also be compatible with the boilerplate's server/manifest.js export, which is a confidence store. To my knowledge the confidence CLI isn't currently used especially often, so I think going with something simple might be a good choice!

TLS cert/key data missing on Store.get(...)

I'm try to use the following TLS object to configure Hapi connections in my manifest. This pulls in a pair of Uint8Array objects, which I'm guessing is the problem.

const tlsOptions = {
    key: Fs.readFileSync('./config/certs/ssl-key.pem'),
    cert: Fs.readFileSync('./config/certs/ssl-cert.pem')
};

Unfortunately, when I extract the manifest using Store.get(...), the key and cert objects are now empty. Is this a bug, or a caveat I'm not understanding? Is there a suggested workaround? Thanks!

Possible way to merge criteria?

I was wondering if there was a way to provide an array for the criteria to filter for? I know the $base and whatever is filtered for via store.get() is merged, but can you merge more than just that by providing an array or something like?

Example...

'use strict';

import * as Confidence from 'confidence';

const store = new Confidence.Store({
    "package-groups": {
        $filter: 'groups',
        $base: [ 'base thingy' ],
        $default: [ 'default' ],
        foo: [ 'baz' ],
        bar: [ 'qux' ]
    }
} );

console.log( store.get('/', { groups: 'foo' }) );
// Result: { 'package-groups': [ 'base thingy', 'baz' ] }

console.log( store.get('/', { groups: ['foo','bar'] }) );
// Result:         { 'package-groups': [ 'base thingy', 'default' ] }
// Desired result: { 'package-groups': [ 'base thingy', 'baz', 'qux' ] }

I realize this is meant for HapiJS, and thats what I use it for mostly, but some of the packages made for HapiJS are pretty useful with just about anything else, such as Joi, as well as Confidence. Im trying to get Confidence working on another (non HapiJS) project, but I need to be able to filter for more than just one thing in the Confidence json object.

I know I could just store.get() multiple times with different filters, and merge the results... But thats pretty messy, id like to avoid that if possible.

Thanks

Add support for environment variables in filters and values

Make confidence isomorphic by leveraging browse crypto if needed

Right now, the Id lib requires the crypto module, I wonder if we could not do the following so one can use confidence in client side app as well:

internals.getRandomBytes = (
    (typeof self !== 'undefined' && (self.crypto || self.msCrypto))
        ? function () { // Browsers
            const crypto = (self.crypto || self.msCrypto), QUOTA = 65536;
            return function (n) {
                let a = new Uint8Array(n);
                for (let i = 0; i < n; i += QUOTA) {
                    crypto.getRandomValues(a.subarray(i, i + Math.min(n - i, QUOTA)));
                }
                return a;
            };
        }
        : function () { // Node
            return require("crypto").randomBytes;
        }
)();

Invalid key names and validation

Hi,

Quoting documentation:

Key names can only contain alphanumeric characters and '_' with the '$' prefix reserved for special directives. Values can contain any non-object value (e.g. strings, numbers, booleans) as well as arrays.

I think this restriction is counterintuitive and severely limits the scope of usage for Confidence while not bringing any clear benefits. In my opinion, there should not be any restrictions on key names, unless it is due to some technical or conceptual limitation.

My example: I want to create a criteria-based configuration for node modules and plugins using their canonical names, but Confidence does not support dashes. And what is actually more concerning, when it checks for key name`s validity, it strips invalid characters and still tries to reach a key using this sanitized (modified) key name, which may result in collisions and unexpected behavior false. So this is actually becomes a 2-part issue:

1. Key names should only be limited by use of '$' prefix and '/' characters, everything else that is valid for JavaScript keys should be valid for Confidence.

2. Joi validation must be used to check key identifiers and exceptions must be thrown. not relevant to the issue

I could help with implementation, but looking at the repo I am not sure if the project is still being maintained.

Any discussion on the topic and counterpoints are welcome.

Thanks for this great module.

UPD: Redacted false statement.

Bring coverage back to 100% after lab fix

Remove module from hapi.js organization

This module doesn't really fit in with the purpose of the org. It was added here because it was developed by the same original team as hapi.

@augnin if you are still interested in maintaining it, I can transfer it to your account or another org.

Feature request

A very useful field would be $override, example:

{
     field: {
        $override: "crieteria",
        $default: "default value"
    }
}

if criteria is passed, then field will get its value, if not it will use the default.

A real case scenario is loading some environment variables and passing them as criterias, if they exist they override the default value.

node.hasOwnProperty is not a function

I have used the following code using node v6

const Confidence = require('confidence');

const store = new Confidence.Store({ key: Object.create(null) });

console.log(store.get('/'));

And it returns the error TypeError: node.hasOwnProperty is not a function.

I have more complex scenarios that also result with the same error. The root cause is if the object does not have Object.prototype as its prototype.

As what I have found, this seems to be caused by some updates on node nodejs/node#7223.
But the change was for query strings only so I am unsure why this was affected.

Probably related as well. nodejs/node-v0.x-archive#9130

Update hapi core modules to namespaced versions.

Hapi modules have been moved into a @hapi namespace on npm. Update package.json to reflect this.

Nicer format for confidence bin output

Please supply a flag or by default output the bin/confidence script to have a more readable format rather than just the plain json dump that may be hard to read if alot of variables are there.

Update Documentation

RangeError with Handlebars

I have this configuration:

var Confidence = require('confidence');
var Handlebars = require('handlebars');

var store = new Confidence.Store();

var config = {
  views: {
    engines: {
      html: {
        module: Handlebars
      }
    }
  }
};

store.load(config);
module.exports = store.get('/', { env: process.env.NODE_ENV });

Which results in an error:

/example/node_modules/confidence/lib/store.js:225
    var keys = Object.keys(node);
                      ^
RangeError: Maximum call stack size exceeded

It is happening because Store.validate tries to inspect the Handlebars module which probably has a self reference. How to handle this issue?

"Invalid node object type" with Regexp

Hi,

I use Confidence to build my webpack configuration. So I need to use a Regexp in the "loaders" part of webpack configuration but this is throwing an error.

Any suggestion how I can use Regexp? And why they are forbidden in Confidence?

Thanks

Support for loaders.

Hey,

I'm not sure where this falls, but sometimes configuration needs file references,
Example can be the tls object of the connection object,

Does it make sense to make Confidence delegate property resolution by loaders ?

For example a config such:

tls: {
  key: "!file:/path/to/file"
}

And delegate to a file loader to resolve the key property on the final object?

The question is whether this falls in the scope of Confidence or not,
At the end Confidence designed to work with a Javascript object, not a JSON, so this sounds like more a job of the manifest reader,

So maybe this falls into Rejoice? I'm not exactly sure.

Thoughts?

$base with arrays is merged with (not overridden by) filtered value

Consider this example code:

const Confidence = require('confidence');
const store = new Confidence.Store();

store.load({ cors: { $filter: 'env', $base: { origins: { $value: 'a' } }, $default: { origins: { $value: 'b' } }, dev: {}, } });
console.log('Expected: b, Actual: ', store.get('/cors/origins'));

store.load({ cors: { $filter: 'env', $base: { origins: { $value: { blah: 'a' } } }, $default: { origins: { $value: { blah: 'b' } } }, dev: {}, } });
console.log('Expected: { blah: \'b\' }, Actual: ', store.get('/cors/origins'));

store.load({ cors: { origins: { $filter: 'env', $base: { $value: ['a'] }, $default: { $value: ['b'] }, dev: {}, } } } );
console.log('Expected: [ \'b\' ], Actual: ', store.get('/cors/origins'));

store.load({ cors: { $filter: 'env', $base: { origins: { $value: ['a'] } }, $default: { origins: { $value: ['b'] } }, dev: {}, } });
console.log('Expected: [ \'b\' ], Actual: ', store.get('/cors/origins'));

store.load({ cors: { $filter: 'env', $base: { origins: ['a'] }, $default: { origins: ['b'] }, dev: {}, } });
console.log('Expected: [ \'b\' ], Actual: ', store.get('/cors/origins'));

Output:

Expected: b, Actual: b
Expected: { blah: 'b' }, Actual: { blah: 'b' }
Expected: [ 'b' ], Actual: [ 'b' ]
Expected: [ 'b' ], Actual: [ 'a', 'b' ]
Expected: [ 'b' ], Actual: [ 'a', 'b' ]

The last two examples don't work as expected.

Note: My current work-around is to move the 'origins' key up a layer, as shown in the third example.

Add bin for generating config stream

Add a binary to allow generating a config from an input confidence file

upgrade to lab 4.x

Please upgrade to lab 4.x.x

upgrade to boom 2.x.x

Please upgrade to boom 2.x.x

Publish on npm as @hapipal scoped package

The next major version of all packages in the org will be published under the @hapipal scope on npm. This module will be published as @hapipal/confidence.

Should we remove $env?

I noticed that $env can be modeled equally well using $param if the user passes process.env in their criteria. I propose that in v6 we remove $env and encourage users to explicitly pass the environment as criteria if that is their intention. I am in favor of this change because it simplifies the API, and encourages the store to not rely on any implicit/global state.

I just made a small alteration to allow $coerce to work with $param to generate this example:

Before (with $env)

const store = new Confidence.Store({
    server: {
        host: 'localhost',
        port: {
            $env: 'PORT',
            $coerce: 'number',
            $default: 3000
        },
        debug: {
            $filter: { $env: 'NODE_ENV' },
            $default: {
                log: ['error'],
                request: ['error']
            },
            production: {
                request: ['implementation']
            }
        }
    }
});

const config = store.get('/');

After (with $param)

const store = new Confidence.Store({
    server: {
        host: 'localhost',
        port: {
            $param: 'PORT',
            $coerce: 'number',
            $default: 3000
        },
        debug: {
            $filter: 'NODE_ENV',
            $default: {
                log: ['error'],
                request: ['error']
            },
            production: {
                request: ['implementation']
            }
        }
    }
});

const config = store.get('/', process.env);

Support for default criteria

It would be useful to be able to define a default criteria.

Currently, when one calls get without a criteria, {} is used by default. I find myself repeating the same criteria every time I have to get a path:

store.get('/a/b', { env: process.env.NODE_ENV });

...

// At some other place
store.get('/c', { env: process.env.NODE_ENV });

If the criteria is more complex, things get more tedious. My workaround is to simply get the root config once and use that everywhere, but that eliminates the advantages of the get function.